| View previous topic :: View next topic |
| Should forums.gentoo.org move to a commercial forums package? |
| Yes -- keeping older posts searchable is more important than sticking with GPL-only products |
|
53% |
[ 84 ] |
| No -- lose some of the older posts and stay on the GPL'd phpBB |
|
46% |
[ 73 ] |
|
| Total Votes : 157 |
|
| Author |
Message |
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
 Posted: Sun Sep 28, 2003 11:55 pm Post subject: |
 |
|
| sonikntails wrote: | My question is what about people who have certain posts bookmarked? Will I loose all of the posts that I bookmark (to read later, to look over, help with future installs/projects) if we decide to start "pruning posts". Thats would seriously suck! And if you "prune" old posts, what about Documentation, Tips and Tricks? Thats a forum that I think should definatly not be pruned (where most of my bookmarks are  ). Just my |
We can selectively prune in certain forums, but the more forums we elect not to prune, the less effective overall the pruning is.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
 |
karlosfwb
n00b
Joined: 17 Mar 2003
Posts: 14
|
| Posted: Mon Sep 29, 2003 2:52 am Post subject: |
|
|
I've used the Invision forums on a small scale, and it seems to be a very quality product. Also, as noted before, you only have to pay if you want "offical" support (though community support is still available), or if you want to remove all branding. Its completely skinnable, and seems to scale pretty well as indicated in this thread:
http://forums.invisionpower.com/index.php?showtopic=73039
_________________
Dulche decorum est, pro patria mori... |
|
| Back to top |
|
|
ddanier
Tux's lil' helper
Joined: 19 May 2002
Posts: 87
|
| Posted: Mon Sep 29, 2003 9:14 am Post subject: |
|
|
| klieber wrote: | | We can selectively prune in certain forums, but the more forums we elect not to prune, the less effective overall the pruning is. |
Pruning is not the "best" solution:
We have the same problems with phpBB at our own website, so we turned pruning on --> First it doesn't delete votes, second sometimes its impossible to go into certain forums because the pune works this long that you get a timeout and sometimes phpBB sends such a long MySQL-Query that nothing happens....
DO NOT USE PRUNING 
Perhaps a good solution would be if some developers in here would help phpBB 2.2 to get ready that we can switch to this then....
(Ok, thats more work than simply buy a license of some other board, but why we all use OpenSource when we don't want to do something for it?)
_________________
My Box said: "Install Win95 or better ..." So I installed Linux. |
|
| Back to top |
|
|
zebbedi
Tux's lil' helper
Joined: 05 Sep 2003
Posts: 123
|
| Posted: Mon Sep 29, 2003 9:38 am Post subject: |
|
|
I started using phpBB on a dedicated server once which received about 8000 unique hits a day, and in the end had to remove it due to it slaughtering mysql. I discovered that apparently its queries arent the most efficient.
I replaced it with a free copy of invisionboard and from that day onwards never once experienced any problems at all even after the hits doubled. I can strongly recommend it as being one of the best messageboards developed. So much so ive been requested to install it for about 20+ sites now. |
|
| Back to top |
|
|
pilla
Bodhisattva
Joined: 07 Aug 2002
Posts: 7729
Location: Underworld
|
| Posted: Mon Sep 29, 2003 12:00 pm Post subject: |
|
|
dumb question. Isn't it possible to use another database manager (ala Oracle) instead of MySQL? Would it mitigate the problem?
_________________
"I'm just very selective about the reality I choose to accept." -- Calvin |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Mon Sep 29, 2003 12:11 pm Post subject: |
|
|
| pilla wrote: | | dumb question. Isn't it possible to use another database manager (ala Oracle) instead of MySQL? Would it mitigate the problem? |
Right now, we rely upon OSU's database server which is far more powerful than anything we own. Thus, we're limited to the database that they use, which is mysql.
If we were to get our own database server, we could install the DB of our choice, of course, and that might help the problem, though that can't be said for sure. If someone wants to write me a check for $5000 to buy the hardware to test it out, I'd be happy to do so. 
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
ian!
Bodhisattva
Joined: 25 Feb 2003
Posts: 3829
Location: Essen, Germany
|
| Posted: Mon Sep 29, 2003 12:14 pm Post subject: |
|
|
| pilla wrote: | | dumb question. Isn't it possible to use another database manager (ala Oracle) instead of MySQL? Would it mitigate the problem? |
It's not mysql that slows forums down. IMHO it's the way how phpBB generates it's SQL-statements. These should to be optimized.
ian!
_________________
"To have a successful open source project, you need to be at least somewhat successful at getting along with people." -- Daniel Robbins |
|
| Back to top |
|
|
Cossins
Veteran
Joined: 21 Mar 2003
Posts: 1136
Location: Copenhagen, Denmark
|
| Posted: Mon Sep 29, 2003 12:29 pm Post subject: |
|
|
What about Zend-cache and that sort of PHP optimizations? Or is it just me who doesn't know what the heck I'm talking about?
BTW, has anyone contacted the phpBB2 team to tell them about our problem? Maybe they have a solution for optimizing the SQL generator. Any way, they would be able to do it for the next release (2.2) so others won't get the same problems...
- Simon |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Mon Sep 29, 2003 12:35 pm Post subject: |
|
|
| Cossins wrote: | | What about Zend-cache and that sort of PHP optimizations? Or is it just me who doesn't know what the heck I'm talking about? |
caching php wouldn't help the inefficiencies in the mysql queries.
| Cossins wrote: | | BTW, has anyone contacted the phpBB2 team to tell them about our problem? |
yes, we've talked to one of their developers a while back. The main suggestions were to a) prune posts, b) disable search and c) upgrade hardware. (d) was to wait for phpBB2.2)
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
idoneus
Apprentice
Joined: 26 Mar 2003
Posts: 243
Location: Graz, Austria
|
| Posted: Mon Sep 29, 2003 2:50 pm Post subject: |
|
|
Well this definately is an important and interesting thread.
First of all, I'm all for real OSS.
I can see that we don't have enough time to wait for a new version of phpBB.
Obviously there is no other real OSS that can meet our demand.
I'm still not comfortable with switching to a non free license. It's not good to give up on OSS, since this is the way we loose freedom.
Deleting posts which are of very very little use, won't free up the database that much, at least I think so.
Deleting / Loosing information is even worse than anything else. So deleting any post in an "important" category would definately be no option for me.
I think that not loosing valuable information justifies the use of a non free licenced software. BUT the second phpBB or any other free software meets our demands, we should emidiately switch back, and support them.
I fear that once everything runs smoothly with the new non-free software nobody will care to evaluate free software.
| klieber wrote: | | If someone wants to write me a check for $5000 to buy the hardware to test it out, I'd be happy to do so. |
I'm not sure if you were serious here. But to me $5000 sound like a sum the community should be able to rise.
Why not post an announcement asking if people would be willing to donate a few bugs. 500 x 10$ would equal $5000. Also mention it in the GWN.
The forums have 29414 registered users, that's 0.17$ or 0.15euros per user. So if every fifth user donates 1$ that machine would be yours.
Maybe I'm thinking of this being to easy, but why not give it a try? |
|
| Back to top |
|
|
legobuff
n00b
Joined: 12 May 2003
Posts: 20
Location: ~ N 41 08.5?? W 96 01.7??
|
| Posted: Mon Sep 29, 2003 7:12 pm Post subject: I'll throw in my $.02. |
|
|
I prefer that we stick with an Open Source solution.
Deleting posts == bad!
Don't do it, period. If staying with phpBB means deleting posts, see above.
Now I keep reading we have ~ a week to fix this. Where did the "week" time-frame come from? What is going to happen after a week is up? Is the server going to spontaneously combust after a week.
If the "week" time-frame is true and the server will explode and we absolutely have to leave phpBB, so be it. Otherwise I would rather stick with phpBB and work on getting what we need fixed in phpBB.
Just my $.02. |
|
| Back to top |
|
|
Techie-Micheal
n00b
Joined: 29 Sep 2003
Posts: 7
|
| Posted: Mon Sep 29, 2003 7:36 pm Post subject: |
|
|
I noticed you guys are still using 2.0.4.
Either 2.0.5, or 2.0.6 fixed some of these problems you are experiencing. |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Mon Sep 29, 2003 7:37 pm Post subject: Re: I'll throw in my $.02. |
|
|
| legobuff wrote: | | Now I keep reading we have ~ a week to fix this. Where did the "week" time-frame come from? What is going to happen after a week is up? Is the server going to spontaneously combust after a week. |
It is a somewhat arbitrary timeframe picked based on the fact that we use someone elses' database server and need to be respectful of their resources combined with the fact that we are spending way too much time feeding and watering the forums as it is, meaning other things are getting neglected.
The forums will not blow up in a week if nothing happens, but waiting several weeks/months for phpBB 2.2 to come out and hope that it solves our problem and/or to magically re-write all the SQL within it to be uber-efficient is simply not an option.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
Blkdeath
Developer
Joined: 29 Jun 2003
Posts: 2
Location: Ontario, Canada
|
| Posted: Mon Sep 29, 2003 7:49 pm Post subject: Re: 4th option? |
|
|
| gaima wrote: | Personally, I'll put my vote to a 4th option.
Make phpBB efficient!
I know enough about PHP and MySQL to write efficient code, and I'm sure there are others here who make my knowledge pale into insignificance.
Why not, in the spirit of all things GPL, give something back? |
I'm really glad you said that, as I was thinking the same thing. We have a plethora of talented developers to choose from (both "official" @g.o developers and users with more programming knowledge than they know what to do with.
I'd really prefer that as a third possible option.
_________________
Have you had your Snerk today? |
|
| Back to top |
|
|
caleb
Developer
Joined: 02 Jun 2003
Posts: 404
|
| Posted: Mon Sep 29, 2003 8:24 pm Post subject: deleting posts |
|
|
Why is this such a big deal? Are there really enough valid posts after 6 months for this to be an issue? I mean, in 6 months time frame most software packages have gone through quite a few revisions, so older threads don't seem to relevant.
Someone please give a *compelling* reason for keeping old posts. |
|
| Back to top |
|
|
Draegonis
n00b
Joined: 29 Sep 2003
Posts: 3
|
| Posted: Mon Sep 29, 2003 8:25 pm Post subject: |
|
|
| Techie-Micheal wrote: | I noticed you guys are still using 2.0.4.
Either 2.0.5, or 2.0.6 fixed some of these problems you are experiencing. |
That would probably be a rather fantabulous idea. Also (apolgies if this has been mentioned before - i'm on the run), why don't you introduce stop-words into your HTTPd, preventing the searching of extremly popular words, which reduces performance drastically? This will be default in 2.2, and i realise that you can't wait that long, but phpbb.com has implimented such a method itself. If you searched for "phpBB" on phpbb.com, without having stopwords introduced.. well, i'm sure you can imagine the result.
Edit: It appears you've got a few MODs giving your touble here too - for one, you aren't automatically forwarded to your posts after hitting submit, resulting in a fair few double postsing here. (Same happens after editing) |
|
| Back to top |
|
|
Draegonis
n00b
Joined: 29 Sep 2003
Posts: 3
|
| Posted: Mon Sep 29, 2003 8:45 pm Post subject: |
|
|
| I also suggest that you guys take a look HERE |
|
| Back to top |
|
|
fca
Guru
Joined: 22 Feb 2003
Posts: 346
Location: Netherlands
|
| Posted: Mon Sep 29, 2003 9:00 pm Post subject: |
|
|
| Draegonis wrote: | | I also suggest that you guys take a look HERE |
As you can see, a certain kurtl has posted in that thread. I wonder who that might be?
As for solutions, I talked this over with a friend of mine, and he said that no there could be solutions in MySQL based forums, but he thought they weren't implemented yet. He suggested hash based searches (whatever that may be) and also that there should be a seperate search server, which could be synced every x minutes. I've also talked a bit with the maintainers of http://gathering.tweakers.net (10 million posts, 70k users) and they really recommended Omega search (by Xapian) and open-source search engine, with a PHP based frontend, to blend it in the rest of the forum.
For what it's worth, I'm starting to learn PHP, but that will take longer than a week (I won't guess how much time a significant improvement on phpBB would take me....) |
|
| Back to top |
|
|
Blkdeath
Developer
Joined: 29 Jun 2003
Posts: 2
Location: Ontario, Canada
|
| Posted: Mon Sep 29, 2003 9:13 pm Post subject: Re: 4th option? |
|
|
| gaima wrote: | Personally, I'll put my vote to a 4th option.
Make phpBB efficient!
I know enough about PHP and MySQL to write efficient code, and I'm sure there are others here who make my knowledge pale into insignificance.
Why not, in the spirit of all things GPL, give something back? |
I'm really glad you said that, as I was thinking the same thing. We have a plethora of talented developers to choose from (both "official" @g.o developers and users with more programming knowledge than they know what to do with.
I'd really prefer that as a third possible option.
_________________
Have you had your Snerk today? |
|
| Back to top |
|
|
A_Jelly_Doughnut
n00b
Joined: 29 Sep 2003
Posts: 2
|
| Posted: Mon Sep 29, 2003 9:36 pm Post subject: |
|
|
This is a known problem with phpBB 2 (as previously noted). Some other things not mentioned in the topic Draegonis linked to are here: http://www.phpbb.com/phpBB/viewtopic.php?t=70559&highlight=search+performance
Also, some optimisations for viewtopic: http://www.phpbb.com/phpBB/viewtopic.php?t=28254
I remember someone had a different setup. They had a perl script index the search on CRON, and a different PERL script to retrieve search results. Not beautiful, but possibly better than what you have.
Another tweak would be to disable the editing of posts. You can also change the length of words that are entered into your database to save space. (Look in functions_search.php for the preg_replace)
It shouldn't surprise you that the phpBB gurus are coming out in full force to save their software from the axe, and I think that we would be willing to work with you to help save phpBB.
A_Jelly_Doughnut
phpBB Support Guru |
|
| Back to top |
|
|
BartVB
n00b
Joined: 12 Mar 2003
Posts: 9
Location: The Netherlands
|
| Posted: Mon Sep 29, 2003 9:53 pm Post subject: |
|
|
I'm running phpBB2 on my site (www.bokt.nl/forums/) which has around 200 users online on average and more than 2M posts, 10k users. No problem at all. My main problem was the webfronted (Athlon 1.6) which was maxing out when we had 300+ users online.
Main problem with plain vanilla phpBB2 is the search system, it takes ages to edit/delete/post a message on boards with lots of posts. Make sure that you have an index on both word_id and post_id in the search_word_match table (you should have on new phpBB2 installs) and if that doesn't help then you can delay indexing to once a day. I've written a fairly trivial mod that only marks a post as edited/deleted/new and then it updates the search tables once a day.
Other (beter) solution would be to write a Xapian frontend for phpBB. I'm planning on doing that but absolutely haven't got the time for that at the moment :\
In short: running phpBB on your site with your hardware shouldn't be a problem with some tweaking.. |
|
| Back to top |
|
|
stuherbert
Retired Dev
Joined: 17 Aug 2003
Posts: 47
Location: Pontypridd, South Wales
|
| Posted: Mon Sep 29, 2003 10:05 pm Post subject: Practical steps |
|
|
Kurt,
I'd strongly recommend all of the following steps:
1a) Upgrade to the latest phpBB version - 2.06. You're running 2.04. Might be better, might be worse.
1b) Upgrade to the latest PHP version, if you haven't already.
1c) Consider which Apache/mod_php combination provides the best performance for serving phpBB.
2) Upgrade the backend to MySQL v4 (if you're not running it already!) and enable query-caching.
3) Check the MySQL logs to make sure that non-indexed searching is *not* taking place
4) Consider rebuilding the database with a different type. Should be some information on the relative performance of each database type on MySQL.com somewhere.
5) Identify the specific SQL queries that are causing the most performance problems, and fix them. De-normalising tables is one approach. Another is to quickly add local caching using memcached. Give me a shout on IRC if you want my help with doing this. Or rope Lisa into adding the memcache stuff - she's probably got as much as experience as me now with that.
Sorry if you've already stated this and I've missed it, but is it only the back-end db server that has performance problems? Is the machine chewing CPU, thrashing too much, or stalling on i/o waits?
Best regards,
Stu
--
_________________
--
stuart@gentoo.org
Gentoo Developer
Trustee, Gentoo Foundation
http://blog.stuartherbert.com |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Mon Sep 29, 2003 11:16 pm Post subject: Re: Practical steps |
|
|
| stuherbert wrote: | | 1a) Upgrade to the latest phpBB version - 2.06. You're running 2.04. Might be better, might be worse. |
Easier said than done because we've hacked up our code somewhat already. I'll see if I can recruit Lisa to help out, though.
| stuherbert wrote: | | 1b) Upgrade to the latest PHP version, if you haven't already. |
running 4.3.2. Will upgrade to 4.3.3 as soon as it gets marked stable in portage.
| stuherbert wrote: | | 1c) Consider which Apache/mod_php combination provides the best performance for serving phpBB. |
Any suggestions here?
| stuherbert wrote: | | 2) Upgrade the backend to MySQL v4 (if you're not running it already!) and enable query-caching. |
4.0.13
| stuherbert wrote: | | 3) Check the MySQL logs to make sure that non-indexed searching is *not* taking place |
we've got indexes galore. By default, (iirc) there was no index placed on the search tables. That was added long ago.
| stuherbert wrote: | | 4) Consider rebuilding the database with a different type. Should be some information on the relative performance of each database type on MySQL.com somewhere. |
innodb on the search-related tables which offers row-level vs. table-level locking. myisam on the rest. innodb hits disks pretty hard, so we didn't use it for eveything.
| stuherbert wrote: | | 5) Identify the specific SQL queries that are causing the most performance problems, and fix them. |
I posted a message earlier about some tweaks that we've made that seem to fix some of the most eggregious problems ("view last 24 hours" link not working, etc.). We'll see what that does.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
BartVB
n00b
Joined: 12 Mar 2003
Posts: 9
Location: The Netherlands
|
| Posted: Tue Sep 30, 2003 6:54 am Post subject: |
|
|
Oh, something else.. Install MMcache Won't help you with they backend but your apache server will love you for installing it
I've altered the 'view my posts' links etc so it only searches posts that are less than a few days old, made this search option a lot more efficient.
If problems still persist you should IMO consider moving towards delayed indexing or take a look at Xapian.
BTW for the upgrades, we do make patches available which work in 99% of the time on modded phpBB installs. |
|
| Back to top |
|
|
Draegonis
n00b
Joined: 29 Sep 2003
Posts: 3
|
| Posted: Tue Sep 30, 2003 6:58 am Post subject: Re: Practical steps |
|
|
| klieber wrote: | | stuherbert wrote: | | 1a) Upgrade to the latest phpBB version - 2.06. You're running 2.04. Might be better, might be worse. |
Easier said than done because we've hacked up our code somewhat already. I'll see if I can recruit Lisa to help out, though. |
Shouldn't be a problem if you upgrade using the UNIX patch styled upgrade that phpBB has released. |
|
| Back to top |
|
|
|
Gentoo Forums Feedback
