As many of you have noticed, the forums have been having significant performance problems lately. Without going into the gory details, most of
these issues have to do with the inefficient way phpBB 2.0.x interacts with the database.

Right now, the apache part of the forums are hosted on OSU's cluster of dual xeon boxes and the database part is hosted by a dual xeon w/ 4GB of RAM, so there is ample hardware behind us. Adding more hardware is not likely to be a tenable solution.

We have a couple of options we're considering, none of which are ideal. One option is to delete older posts and/or move them to an "archive" section which would be unsearchable or really hard to search (which is, imo, effectively deleting them because nobody is going to sift through all that)

Another option is to migrate to an alternate forums package. Specifically, vBulletin, which has proven to be quite scalable and effective with larger sites. The main drawback to this is the fact that it is a commercial software package, which tends to raise the ire of some of the more zealous OSS folks out there.

So, my question to you is, how would you feel if we were to do this? Horribly offended? Saddened because there isn't an OSS program to meet our needs, but pragmatic about the need to do *something*? Or C, none of the above?

Basically, the choice comes down to one of two things:

1) Lose some older posts and stay on a GPL-compatible package
2) Move to a commercial solution and keep all the posts.

Note that vBulletin *is* open source in that you have access to the source code and can modify it. It's just not Free.

Just as a follow-up, another commercial package we would look at would be Invision, which has also received good marks on speed. However, it's still a commercial package, so the questions/decisions above remain the same.

--kurt

klieber wrote:So, my question to you is, how would you feel if we were to do this? Horribly offended? Saddened because there isn't a GPL'd program to meet our needs, but pragmatic about the need to do *something*? Or C, none of the above?

If there is no Free forum which can handle it, I think it would be acceptable to use something else, as long as it's not proprietary. However, this raises 2 more important questions:
1. Can phpBB2 databases be converted to the new database format?
2. Can the new database format be converted back should phpBB2 improve their performance?

klieber wrote:Basically, the choice comes down to one of two things:

1) Lose some older posts and stay on a GPL-compatible package
2) Move to a commercial solution and keep all the posts.

Summarized in #2. Also note that commercial does not mean it cannot be GPL'd.

klieber wrote:Note that vBulletin *is* open source in that you have access to the source code and can modify it. It's just not Free.

"PHP-based" implies the first sentence.

klieber wrote:Finally, to keep this process somewhat manageably, I ask that you make your responses on the forums. I've also set up a poll there so we can vote on what should be done. I will likely limit my responses to that discussion.

Bleh now I need to convert this to the forum format. =p

Hello. I have not voted as yet as I'd like your view on a certain portion of the gentoo linux social contract. How does migrating to a commercial package relate to the above or does this only refer to and apply to the physical operating system?

Narada wrote:How does migrating to a commercial package relate to the above or does this only refer to and apply to the physical operating system?

It's certainly a gray area, and it's one of the reasons I posted my original question -- to solicit feedback from all of you.

--kurt

klieber wrote:Another option is to migrate to an alternate forums package. Specifically, vBulletin, which has proven to be quite scalable and effective with larger sites. The main drawback to this is the fact that it is a commercial software package, which tends to raise the ire of some of the more zealous OSS folks out there.

I assume you've already looked into the other GPL'd or free (money-wise) forums packages available, and all fall short?

Luke-Jr wrote:1. Can phpBB2 databases be converted to the new database format?
2. Can the new database format be converted back should phpBB2 improve their performance?

1. Yes, vBulletin and Invision both offer scripts to convert from phpBB2.
2. Yes, phpBB2 offers scripts to convert from vBulletin. There is currently no supported Invision --> phpBB2 migration, but since all the data is stored in mysql, it's simply a matter of parsing/rearranging it to suit the different BB systems.

--kurt

spyderous wrote:I assume you've already looked into the other GPL'd or free (money-wise) forums packages available, and all fall short?

Yes, most of them, at least. It's hard to cover them all. We didn't look at anything that ran on java, simply because we don't have the in-house expertise to manage and support that.

There's a list of larger BB installations. It's not complete, but it does give you a decent indication of what the big sites run.

--kurt

Personally, I'll put my vote to a 4th option.

Make phpBB efficient!

I know enough about PHP and MySQL to write efficient code, and I'm sure there are others here who make my knowledge pale into insignificance.
Why not, in the spirit of all things GPL, give something back?

Mike

gaima wrote:I know enough about PHP and MySQL to write efficient code, and I'm sure there are others here who make my knowledge pale into insignificance.
Why not, in the spirit of all things GPL, give something back?

The phpBB team is, in fact, working on version 2.2 which is supposed to solve a lot of these problems. However, there is no release date for 2.2 and we're at a point where we need to do something now, rather than wait for 2.2 to be released and hope that it scales better.

--kurt

klieber wrote:

Narada wrote:How does migrating to a commercial package relate to the above or does this only refer to and apply to the physical operating system?

It's certainly a gray area, and it's one of the reasons I posted my original question -- to solicit feedback from all of you.

Having read your reply here is my view. If the extract of the Gentoo Linux Social Contract quoted in my previous post can be taken to apply to all aspects of Gentoo Linux then switching to a commercial package may be a clear violation of the contract. This is because then a fundamental portion of Gentoo Linux that is critical to its survival will depend upon a piece of software that may not fulfil the criteria listed in the contract.

On the other hand, the issue of migration is being raised in good faith with good reason and has been very well justified and therefore I wholeheartedly support the move to a better package. Inversatility and inflexibility only result in stagnancy. However, migrating to a commercial package with the current contract in place is a contradiction and so if the adoption of a commercial package is decided upon then formal justifications of the violation of the current contract must be published and the contract modified to reflect the reasons behind the need for change. This strikes me as quite a drastic and core alteration and may require careful consideration as to whether a proposal that threatens the current contract is worthy of adoption.

I appreciate the difficulty of this issue and can see why so much attention has been drawn to it. May it be an idea to approach VBulletin and Invision to ask whether they are prepared perhaps to make a donation in the form of use of their software or even a favoured discretion on the cost or even look for ways that the new packages may be able to fulfil the criteria of the contract?

IMHO the two-forums idea is almost as bad as deleting posts in some senses. There are already enough people who don't search, and forcing them to search in two places... pretty much a hopeless effort. If you do do that though, at least make it easily searchable, becuase there are also many people who do search and like having over a year and almost 30 thousand peoples advise at their fingertip. Like you said the forums are a valuable archive of information. You mentioned that you didn't look at java based solutions because there was no one to maintain them. I'm sure there's got to be someone in the gentoo community willing to do that. If there are any java based solutions that scale well and are open souce(in the social contract sense) why not toss a mail onto gentoo-dev to see if there's anyone who would be willing to set up and maintain it? If not, then imo we should defenitly move to another package such as vBullitin which would scale well. Let's not drag our way through solutions that don't work out solely because of idealistic reasons.

Narada wrote:May it be an idea to approach VBulletin and Invision to ask whether they are prepared perhaps to make a donation in the form of use of their software or even a favoured discretion on the cost or even look for ways that the new packages may be able to fulfil the criteria of the contract?

We've already had one member offer to purchase the commercial forums package of our choice for us, so getting the software for free is not an issue. If we do get the software donated, however, does that mean we're in compliance with our social contract?

Also, I personally (and you are free to disagree with me) believe that "Gentoo Linux" is separate from the infrastructure that supports it. While we've been very fortunate in getting donations and sponsors, we have had to purchase hardware in the past in order to support development efforts. We also use UltraDNS for our DNS services. (though they do donate it to us at no charge.)

Finally, I feel that deleting older posts (or making them inaccessible via searching, which is, in effect, the same thing) carries a heavier price for our user base than making use of a commercial, open source (but not Free) product. There is literally one checkbox in phpBB2's admin control panel that would make all our troubles go away by pruning older posts. However, it is because we recognize the value that these posts hold in the form of knowledge, experience and helpfulness to our user base that we're exploring all possible options to keep it intact.

--kurt

klieber wrote:

spyderous wrote:I assume you've already looked into the other GPL'd or free (money-wise) forums packages available, and all fall short?

Yes, most of them, at least. It's hard to cover them all. We didn't look at anything that ran on java, simply because we don't have the in-house expertise to manage and support that.

There's a list of larger BB installations. It's not complete, but it does give you a decent indication of what the big sites run.

--kurt

From that list:
Gaia Online, ranked #3, has more than 4 million posts and 80,000 members. Gentoo, ranked #50, has about 540,000 posts and 30,000 members.

But Gaia, with eight times more posts and 2-3 times as many members, seems more responsive than Gentoo forums.

Perhaps it would be worthwhile to ask Gaia Online, another phpBB-based forum, how it keeps things running so well. I checked it out and its response seemed pretty good.

The same applies to the other phpBB-based forums ranked above us.

spyderous wrote:Perhaps it would be worthwhile to ask Gaia Online, another phpBB-based forum, how it keeps things running so well. I checked it out and its response seemed pretty good.

Already did that. The admin deletes posts that haven't been responded to in over a month. If we did the same, our forums would fly, but lots of archived data would be lost.

The 4.5MM posts is a misnomer, btw, as the post count does not decrement when older posts get pruned.

--kurt

This thread has made me analyse, closer than ever before, my feelings about open-source software.

I started out writing this post in regard the commercial co$t of the proposed vBulletin software - the vBulletin site suggests we're only talking in terms of a hundred or two $$$ (depending on licensing requirements & decisions); I appreciate this is a trivial amount, however it just sticks in my graw a little.

I started to say that personally I wouldn't mind if the software was closed-source and only distributed as binaries, if it were free beer & clearly the best tool for the job.

But the thing is that open-source software is JUST better. It's not just the software - it's the model. I'm skint as sin right now, but if it's a matter of money, I'd rather put my hand in my pocket & give it to the developers of phpBB, if that's OSS. Because then that software belongs to all of us. Forever, for free.

I guess it IS a matter of principle for me - the money that could be spent on a commercial product could better be spent supporting an OSS product. If someone is writing Free (tm) code, then that individual is behaving more ethically than any commercial organisation, IMO. I believe in supporting that effort.

Would Gentoo users really mind if it even cost more..? Without a doubt, the Gentoo forums benefits our community ENORMOUSLY. Could this now be a case whereby Gentoo can start to support *other* community-based projects..?

If all the regular forums users contributed, I daresay we could raise quite a reasonable sum - I really think this would be better contributed to the phpBB developers.

klieber wrote:Also, I personally (and you are free to disagree with me) believe that "Gentoo Linux" is separate from the infrastructure that supports it. While we've been very fortunate in getting donations and sponsors, we have had to purchase hardware in the past in order to support development efforts. We also use UltraDNS for our DNS services. (though they do donate it to us at no charge.)

Klieber. Having read your reply the matter is considerably clearer in my mind. I have now made up my mind and have voted to say that keeping old posts is more important than using GPL'd products since that was most synonymous with moving to a better product. Yes, I agree full heartedly that the incoming infrastructure supporting Gentoo Linux can indeed be held to be separate from the outgoing physical product and also that unrestrained and novel efforts must be continually made to keep the infrastructure robust given the size of the user base. In that perspective, the social contract remains unbreached. In fact, if I remember back to the blissful night that I discovered Gentoo and read the social contract I did interpret it as applying to the operating system but wasn't quite sure if it really did officially apply exclusively to it. As such, I support the move to a more efficient product which if made shall no doubt eventually be much appreciated by the user community. As regards which one, recording performance benchmarks between products is indeed a good idea however I'd recommend against picking one for which support isn't currently available internally to set up and maintain.

strolls wrote:If all the regular forums users contributed, I daresay we could raise quite a reasonable sum - I really think this would be better contributed to the phpBB developers.

And if that happens and they come out with a version that scales to the levels we need it to, we would certainly look at using that.

The problem is there is no release date for 2.2. They've reached milestone status, but there's no telling when the final version will be released. We don't have the luxury of time -- we need to do something with a timeframe measured in days or weeks, not one measured in months.

--kurt

As you all know, one of the greatest benefits of Gentoo is the incredible community. This brings up two thoughts in my mind:
1. If the forums are "unsearchable" and older posts get lost, it is nearly pointless (because atm, most Gentoo-related problems can be solved be searching the forums).
2. Wouldn't it be possible in a such community to raise enough brainpower to help the phpBB people improve their product? So many great hackers, programmers, and potential bug-testers could easily improve on phpBB drastically. IMO.

- Simon

Well, at least vbulletin isn't that expensive. However I really like phpBB2. Any thought of installing the CVS version of phpBB2 2.2?

Or just purging the keyword search table and reoptimize the database?
http://www.phpbb.com/phpBB/viewtopic.ph ... word+table

The only thing I've thought of I haven't already seen psoted is to put half the forums on one set of hardware and the other half on another set of hardware, provided you have the hardware. Then cross your fingers that phpBB 2.2 comes out in time or what have you. That, or use google to search old posts, while new posts are maintained in the index table.

It looks like Invison Board does not cost any money unless you want technical support from them or want to remove their copyright notice. I'm not at all suggesting that Invision is Free with a capital F, but I thought this worthy of mention.

Here are the license agreements for both of the boards that klieber has mentioned:
Invision Power Board License Agreement
vBulletin License Agreement

Someone mentioned that we could avoid the whole moral idea by having the kind folks at vbulletin donate their software to Gentoo.

Would it be the same if a someone else were to pay for the license for Gentoo to use? I'm willing to pay for the initial license and the follow-up licenses as well for as long as the Gentoo webmasters want to keep it.

So technically, it's being donated by me for an indefinite amount of time. Would that count?

I think it's far more important to keep everything running smoothely than to split everything up, cause mass confusion, and possibly make it inconvenient for regulars and newcomers alike to get access to the information that they need.

I've been using Gentoo for the longest time now, and I've yet to give anything back. Here's my chance, and I hope everyone can agree with *the cause* rather than turn their nose up to the *effect*.

Ryan

if phpBB has a 'crap' method of database handling then why don't the developers just change it to something better sadly i cannot help do this yet but IMHO phpBB is the best forum and all others are just horrible by comparison in looks and features phpBB is superior guess the database just needs fixing

stonent wrote:Well, at least vbulletin isn't that expensive. However I really like phpBB2. Any thought of installing the CVS version of phpBB2 2.2?

Or just purging the keyword search table and reoptimize the database?
http://www.phpbb.com/phpBB/viewtopic.ph ... word+table

The last thing we need is BETA software to make things worse.