Migrating to a commercial PHP-based forums package

Message

Koon · Post by **Koon** » Thu Oct 02, 2003 7:51 am

My 0.02$ :

Few will search on a separate forum for old posts. Moreover, having too much information makes it unsearchable... I think you should reduce total mass of posts to critical size :

- Auto-prune opinion forums (Gentoo chat, Off the wall) and administrative (Duplicate posts) to kill > 6 months-old posts
- Auto-prune *all* forums to kill > 1 year-old posts
- Create a separate phpBB for International forums ?
- Switch to phBB2.2 when available, shall it solve anything

-K

shdwrnnr · Post by **shdwrnnr** » Thu Oct 02, 2003 8:27 pm

A copule of questions, which I think are important:
1. Would excluding posts older than X days from the search words table rather than pruning the posts provide increased performance? This could be used as a temporary solution until phpBB is improved. Once phpBB was improved, the older posts could be added to the search words table once again -- they wouldn't be lost forever.
2. Is there any estimate on how far back we would have to exclude/prune posts before performance would be improved to an acceptable level?

At this point, I think I'm in favor of making some of the older posts unsearchable (but not deleted altogether) for a while until things are sorted out with phpBB. I think it's something everyone can live with for a month or two. I also think that dealing with searches only 6 months back (or whatever) for a little while is better than having to switch to an altogether new system. Not only would the Gentoo devs have to take more of their time to get this done, but all users of the forum would have to take a little bit of time to adapt to the different system.

I prefer to think of this as a minor inconvenience of not being able to search further back than a couple of months rather than a situation where thousands of posts will go to /dev/null forever. I'm guessing the devs should be able to get full searching back in a few months. With how fast Gentoo moves, most of the relevant information is only weeks old, not months or years, which is why I personally consider it only a minor inconvenience.

Retriever II · Post by **Retriever II** » Fri Oct 03, 2003 1:27 am

Well, I'll just throw in my two cents. I would support a move to another software sollution if it meant having or not having the old posts at our fingertips, as I know I've dipped into them several times before creating new threads. Especially Invision Board, as I've been part of their community for several years, but that's more of a personal reason than anything else.

I know it's been mentioned Invision is Free (foregoing official support), though not Open Source by any means. For that I like it a bit more than vBulleton.

Anyway, whatever happens, I hope that maintaining the extensive archive of posts should be held priority. One of the things that makes Gentoo so wonderful is the extensive wealth of information on these forums, and 9 times out of 10, I've been able to search and find the answers I've needed to problems. I'd hate to see that go.

rsk · Post by **rsk** » Fri Oct 03, 2003 6:27 pm

look at all these people throwing in their two cents... if this keeps up, you'll have your $5000 server in no time!

I think you should rewrite the forums in Java using J2EE design patterns... and then next year when you're done, we'll be ready to rock!

I liked the optimization suggestions that Burt provided (especially the query caching on the back end). Are the SQL comments defined clearly someplace that can be edited, or are they created on the fly from a generator?

Can the new "optimized" code from 2.2 be backported for 2.0.x in this case? Or was there too much supporting work that would be backported as well?

I don't mind you pruning pointless forums, but leaving things alone like the desktop forum, tips and tricks, etc. is what we don't want to loose. I don't think anyone would care that you pruned year old discussions about september 11th from the off-topic forums...

Andersson · Post by **Andersson** » Sat Oct 04, 2003 1:40 am

So... "next friday" is over, at least over here so anyway, tell us what happened! Did we erase all the posts? Or did we trade our souls for commercial software?

Post by **pjp** » Sat Oct 04, 2003 2:00 am

The decision hasn't been made yet. "Next friday" wasn't meant to be taken literally. The point was being made that we can't wait until some unkown point in time (phpbb 2.2 release date) before making a decision.

Some changes have been made, and we have someone working on upgrading from 2.0.4 to 2.0.6.

shawk · Post by **shawk** » Sat Oct 04, 2003 9:02 am

Going through the post at http://www.phpbb.com/phpBB/viewtopic.php?t=139969 where Kurt had a little exchange with the phpBB Lead Developer I am a little irritated at this guys attitude towards people using his software. Not that I want Vip treatment here, but this forum is not some 20 000 posts thingy. Also we made it clear that we would stay with phpbb if possible.

Unfortunately so far nothing is inidcating that phpbb2.2 will scale up the way we need it to, apart from some very vague hints from the Lead guy. Maybe it'd work for another half a year, but after that we might hit the bottleneck again.
It also looks like that the 2.2.0 release will most likely be a real .0 release with many new features and be bugged to some degree which doesn't really help either.
From what I gather this issue with our forums needs to be adressed quickly before they bog down completely and waiting for unreleased software should not be the first option.

Our forums are one of the biggest resources on linux problems and topics on the net. Even though we are gentoo specific I cannot recount the times I have sent other people here to get info on crypto, printing etc. in older posts.

Being able to search the whole database is of utmost importance to me, and many others. This is like a library, a huge one. It's one of the key reasons people will stick to gentoo, because they can find help here, on almost anything.

I am one of the pragmatic guys. Use OSS wherever possible. But there are times when one has to weight whats more important, what the real aim is:
Are we running a forum for Gentoo and want it to be as effective and efficient as possible, to work as its intended to or
Do we want to promote OSS to the point where we sacrifice functionality that is certainly of huge benefit to the gentoo project and many others?

Personally, I'd like to see those forums retain their ability to search the whole database.

pilla · Post by **pilla** » Sat Oct 04, 2003 2:15 pm

I've seen a great deal of improvement in the search for the last days (well, know I am able to search again).

I don't wanna put more stress than necessary in the sysadmins of our site, and a migration to another software package will not be smooth IMO. Thus, if we are able to stick with phpBB it would be a great thing.

BUT if the sysadmins feel that it cannot be done with the current package, I feel that keeping all the knowledge we accumulate in these forums is more important than sticking with a certain sw package.

With all this blah-blah-blah, I am just trying to say that I would support the sysadmins' option, whatever it would be.

SHS` · Post by **SHS`** » Sat Oct 04, 2003 2:51 pm

I'm not here on these fora enough to know how well the initial set of changes have effected performence, but seeing at it's outwardly still running 2.0.4, there should be more improvements once 2.0.6 is in place with whatever other changes regarding inprovements in search indexing efficiency are put in place.

However, I did spot this post and I think it's more a testiment how phpBB has managed to scale:

klieber wrote: If we split the db onto two separate servers (hardware which we don't have, btw) I'm not sure that's going to solve the primary problem, which is people searching the forums and hitting the 1.5GB wordmatch table. We might gain some additional room to breathe, but I don't think it would solve all our problems.

--kurt

Now this suprised me, as in both phpBB2.x and vBulletin2.x, their respective search indices are of similar size to the posts table, so that would mean the post_text table here at Gentoo would also be about 1.5GB (+/- 20%), putting the toal DB size about 3GB.

Having been familiar with large phpBBs and vBulletins, I know that 1GB of database typically equates, on average 1.5 million posts... so the 3GB DB of these forums would be close to a typical forum of 4 to 6 million posts.

Also, even with vBulletin, search is an issue with forum performance, and on their support forums big boards (1M+) either needing more hardware, or deviate from the core codebase so they could use InnoDB more effectively.

Looking here: http://www.big-boards.com/ ... you can see with the largest vBulletins and IPBs, search is disabled for guest users, yet all the big phpBB's it's fully functional for guests too.

So if you wish to convert to another forum package, that's fine, though I think to not use an OSI solution is rather against the Gentoo Social Contract... and lastly, I'm quite sure that the two suggested alternative will function no better, since the Gentoo fora is actually a lot bigger than it's postcount suggests.

rsk · Post by **rsk** » Sat Oct 04, 2003 4:29 pm

Hey hey, whatever you guys are doing to the forums is working. Its not acting like a dead mule anymore.

Keep up the good work!

Techie-Micheal · Post by **Techie-Micheal** » Mon Oct 06, 2003 1:18 pm

shawk wrote:Not that I want Vip treatment here, but this forum is not some 20 000 posts thingy.

And neither is www.phpbb.com/

Unfortunately so far nothing is inidcating that phpbb2.2 will scale up the way we need it to, apart from some very vague hints from the Lead guy.

"Basic" search functionality is enabled at area51, and not yet in CVS, so hold on to your hats.

The problem is, as noted earlier, is that we (www.phpbb.com) run significantly "less" hardware, and yet do not have the problems that you guys seem to have. Something just doesn't jive here. I also noticed a couple of MODs here, and those may also play some part.

It also looks like that the 2.2.0 release will most likely be a real .0 release with many new features and be bugged to some degree which doesn't really help either.

Not to be rude or anything, because I quite like seeing a large community like this using phpBB, but how do you know this? phpBB 2.0.x had significantly less people testing CVS, and commenting on things than does 2.2.x ... As I am sure you well know, the wider the testing base, the more different hardware (configurations) you will run in to, and the more you can develop fixes for them.

From what I gather this issue with our forums needs to be adressed quickly before they bog down completely and waiting for unreleased software should not be the first option.

Definitely. Which is why you guys should update to 2.0.6. It really bothers me to see someone complaining about a software package, and yet they don't use the latest version, which may indeed solve their problem(s) ...

Do we want to promote OSS to the point where we sacrifice functionality that is certainly of huge benefit to the gentoo project and many others?

Again, as noted before, there are other sites as large, or maybe even larger than yours, and they are not having a bit of trouble. There are discussions on www.phpbb.com for people running larger boards, and many of the changes suggested have helped a large number of people.

Personally, I'd like to see those forums retain their ability to search the whole database.

Follow our 3 step plan.

Sorry, couldn't resist.

Seriously though, there have been things suggested that haven't been done yet, for whatever the reason.

bzugda · Post by **bzugda** » Mon Oct 06, 2003 4:26 pm

So where are we with this? Seems to be 3 days since the people in the know commented on the future of my beloved forum. I noted Rsk observing that the speed has picked up but I've not really been on enough recently to comment. Even if that is the case, I'm curious to hear whether or not sufficient tweaks are being made or are we waiting for the commercial software to arrive.

Anyway, The Gentoo forums are imho the greatest source of Linux info on the web with an excellant community of users and a wealth of technical information. I don't think we can afford to lose any of that information so do what needs to be done to preserve it.
That said, the feeling from the PhpBB people who've posted seems to be that we /shouldn't/ be having a problem.
So, What's happening dudes ?

rsk · Post by **rsk** » Mon Oct 06, 2003 4:29 pm

yae, what bzugda said.

klieber · Post by **klieber** » Mon Oct 06, 2003 4:31 pm

Techie-Micheal wrote:Definitely. Which is why you guys should update to 2.0.6. It really bothers me to see someone complaining about a software package, and yet they don't use the latest version, which may indeed solve their problem(s) ...

As already mentioned previously, this is underway at the moment.

There are discussions on www.phpbb.com for people running larger boards, and many of the changes suggested have helped a large number of people.

Many of those "suggestions" involve significantly hacking up the phpBB source code which is not a "solution" IMO as it indicates problems endemic to the product itself. Also, many of those larger boards prune posts and/or disable searching. We already know both of those options will solve our problem. The question is whether or not there are other solutions that will be less drastic.

The point of this thread was never to bash on phpBB. It has served us well up to this point. It was merely to identify a problem, discuss potential solutions for it and let the community decide which one to choose. I wish people would stop being defensive, stop pointing fingers and instead focus on fixing the problem at hand. (this goes for both sides of the argument-that-never-should-have-been-in-the-first-place)

As an update, we've made some changes as suggested by BartVB and some of our own developers that seem to have helped. We still have an odd issue with search queries hanging around for hours on end that we're trying to track down, but at the moment, we're in a holding pattern. We're not at the point where I can say things are "fixed" but we're at least limping along well enough so we can delay a decision on migrating to another forums software package.

--kurt

Techie-Micheal · Post by **Techie-Micheal** » Mon Oct 06, 2003 4:44 pm

klieber wrote:
Techie-Micheal wrote:Definitely. Which is why you guys should update to 2.0.6. It really bothers me to see someone complaining about a software package, and yet they don't use the latest version, which may indeed solve their problem(s) ...
As already mentioned previously, this is underway at the moment.

Fair enough.

kurt wrote:
There are discussions on www.phpbb.com for people running larger boards, and many of the changes suggested have helped a large number of people.
Many of those "suggestions" involve significantly hacking up the phpBB source code which is not a "solution" IMO as it indicates problems endemic to the product itself.

What makes this even more bizarre for me as a phpBB Support Team Member, is that www.phpbb.com has made absolutely no changes to the phpBB code, and they run without problems. I just find that really weird.

kurt wrote:We still have an odd issue with search queries hanging around for hours on end that we're trying to track down, but at the moment, we're in a holding pattern.

Out of curiousity, how often do you OPTIMIZE your tables? The other thing I am curious about is how many slow queries you have. As I mentioned above, something just doesn't feel right. There are large boards out there (other than www.phpbb.com) that AFAIK, have not made any changes whatsoever to the source code, and they run just fine, as we do. It is just something that bugs me.

mmealman · Post by **mmealman** » Mon Oct 06, 2003 5:42 pm

It shouldn't surprise you that the phpBB gurus are coming out in full force to save their software from the axe, and I think that we would be willing to work with you to help save phpBB.

The above is why I always end up back to using OSS software. When I'm running a commercial package, heck any package, something always goes wrong somewhere. But with commercial packages they truly don't give a crap about you, you're just $xxx that's already in their pocket, and you can't fix the problem yourself because the source is locked up.

But OSS projects don't deal in $xxx, they deal in reputation. So it's an ego thing to keep you running on their software. And even then you can usually hack up your own solution to work around the issue, because the software is totally open to being modded.

Switching to a commercial bulletin board might be a short term fix, but working on improving the situation with existing OSS software is a long term solution.

If the current software absolutely ends up having to be pruned, why can't we prune the old posts out to a seperate archives server that can store this stuff for years?

Put a command line php script that crons nightly or weekly and have it move out the older posts to another server in a solid search-only format.

Don't have that 3rd server to use for archives? I'm guessing we could do a quick donation drive to raise the $500 bucks it'd cost to buy the thing: http://www.aaronix.com/catalog/product_ ... ts_id/2133

Lucero del Alba · Post by **Lucero del Alba** » Tue Oct 07, 2003 1:51 am

Hello people. I didn't follow the whole discussion (Ok, maybe I should) but some replies and the original post. This is what I think:

In my opinion, if we do believe in Free Software, in the GPL sense, (personally, I really do) and have a problem with a piece of it, we -as a community- should work on improving this software (call it phpBB, or whatever) and to share this improvement with the rest of the people out there. It is no always clear if we can do this with all the proprietary options under their respective licenses.

Maybe what we may wonder is if we do believe in Free Software... perhaps we don't, but I think that we do it.

Cheers.

georwell · Post by **georwell** » Tue Oct 07, 2003 4:25 am

I think we should stick with phpBB until the upgrades are completed. It might also help to list the specific problems like the search queries staying around. You should also change the default search from all posts to something like six months. That might help a bit.

Thanks for working on the forums. They are the best documentation for any linux distro.

lars_msh · Post by **lars_msh** » Tue Oct 07, 2003 7:25 am

Sorry for being a bit late to the party...

One of the factors that attracted me to Gentoo was the existence of a phpBB forum for support. In fact, it is such a great resource I come here for general Linux queries in preference to Google.

I don't think it's a good idea to rush through any changes. Deploying an alternative forum solution is not a small task, nor is migrating all the users and topics. While short term performance problems are unpleasant, focus should be maintained on the best long-term solution.

My view on pruning topics is that this is not a disaster. Perhaps a backup can be made in case somebody feels like trying to restore them later, but to be honest there are advantages too - it will make it easier to search for current information.

As already discussed, moving away from a GPL software solution is not to be taken lightly, although it's not the only option. To use an analogy, nobody would like to see it recommended that a heavily-loaded Gentoo box should be replaced with Solaris, although obviously there are always cases for using some particular tool for the job.

My own reasons for going with phpBB was that the non-free forums software usually involves paying a licence fee and being required to place a link to the vendor on every page. More seriously, if you go with non-free software, then you make it difficult for the Gentoo community to help out with any performance or feature enhancements in the future.

I haven't yet exammined the phpBB 2.2 features for database performance, but my experience of phpBB is that any gains are likely to be outweighed by new features. It's probably better to optimise the 2.0.x series for Gentoo's specific performance requirements, and not to count on support from the phpBB team in the area of scalability.

I'm not sure if it will help, but if your web server has spare CPU cycles, I wrote an XMLCache phpBB module that will try to reduce the number of SQL queries. Admittedly it only does the 'easy' queries, but it knocks off at least 3-4 queries from every single page view.

http://www.ukmix.org/about/phpbb/xmlcac ... 0-beta.zip

I never got around to getting a 1.0 release put up on phpBBhacks.com but it's been in production use on my heavily-customised phpBB installation at http://www.ukmix.org/forums/ for more than a year. Admittedly the forums are less busy than Gentoo, but I have to use limited shared server resources...

I'm willing to have a good look at helping to optimise phpBB for Gentoo if you want (did you do that stop words trick btw?), or if it comes to it I'll be the first in the queue to donate towards the cost of new hardware.

baudelaire98 · Post by **baudelaire98** » Tue Oct 07, 2003 9:04 am

Edweirdo wrote:What if you were to move the forums to a USENET news server? It should be able to handle the load and if you connect it to the rest of the world Google (and the other search engines (are there others?)) would automatically index it for searches. Others who are non-gentoo people might see answers to general questions where they otherwise wouldn't.

Just my idea.

Hear hear!
Just what I've been thinking trying to search these graphics bloated forums.

charlieg · Post by **charlieg** » Tue Oct 07, 2003 10:29 am

Why not use something that isn't PHP?

PHP does not have performance as one of it's better attributes. Hell, even a Python/Psyco combo would be infinitely better in that area than PHP. However, I'm sure there is a C-based BB out there. There must be. I mean, BBs are a concept that is much older than the recent scripting languages, right?

charlieg · Post by **charlieg** » Tue Oct 07, 2003 10:36 am

A quick google or two later, and I found tForum - although it's groans PHP as well. (I have a dislike for all things PHP, I don't know why.)

Anyway, one of the comments I found on it was, "Blazingly fast" which is what we want, right?

And their frontpage marketing:

Speed and Scalability
Recent improvements in both parsing and table optimization have made the board even more speedy and efficient. The forum allows for an unlimited amount of boards, topics, messages, and users. It won't matter if your needs are few or great as this forum will provide the answer.

Anyway, there's more boards listed here for those interested.

charlieg · Post by **charlieg** » Tue Oct 07, 2003 10:38 am

Ooo... Phorum sounds nice too.

This was my favourite feature: mailing list integration - how funky would that be?

blcknight · Post by **blcknight** » Tue Oct 07, 2003 3:57 pm

I'm a little disturbed by some of the posts here equating Open Source Software with being gratis. After reading through the vBulletin license argreement I wouldn't say that it qualifies as "open source" by the OSI standards and I believe that switching to vBulletin wouldn't fit with Gentoo's social contract and general ideals. However, not all software needs to be free as in beer. Those who work deserve their pay. I personally disagree with vBulletin's model as how to charge for software. There's far better and more open ways to charge for software, especially for a product based off of two other GPL products -- PHP and MySQL. It can almost be equated with biting the very hand that feeds you.

Someone who takes and uses open source software and then turns around with their derivative product and slaps on a restrictive license has no place in the open source community. I realize that much work went into the creation of the software, but far more went into PHP and MySQL. A better business model for vBulletin would be to charge for the medium the software is on -- i.e. a CD and include some detailed manuals. However, no restriction is placed on the source code itself. GPL.

As far as Gentoo switching to a quasiopen source product, I disagree in this specific case. Especially since they're license is so restrictive and it's based off other GPL products. To use that software would validate their business model. I'd be more willing for Gentoo to switch to a Windows machine with proprietary closed-source BB software! At least they have no delusions about being open source or based on it.

There are several alternative options which have been already proposed:
- locate the bottleneck in the phpBB code. fix it.
- delete posts with zero responses, delete posts with zero historic value ("i.e. i hate gentoo! i can't get it to install! i hate linux! i'm going back to windows!")
- two seperate phpBB servers, one with the old "pruned" messages (read-only). i'd bet it's a trivial source code modication to the pruning code to have it export/move it to a different database.

nianderson · Post by **nianderson** » Tue Oct 07, 2003 4:59 pm

Perhaps it would be worthwhile to ask Gaia Online, another phpBB-based forum, how it keeps things running so well. I checked it out and its response seemed pretty good.

The same applies to the other phpBB-based forums ranked above us.[/quote]

http://ian.go-gaia.com/forum/viewtopic.php?t=153918
Starting tomorrow we will start to delete old posts within large topics. On all topics that are affected, the first week of posts within the thread will remain, while anything inbetween then and last month will be deleted. This change will help speed up Gaia even more.

seems they are taking steps to deal with speed issues

Migrating to a commercial PHP-based forums package

Should forums.gentoo.org move to a commercial forums package?

my personal point of view

My views

OSS != Free

Re: Migrating to a commercial PHP-based forums package