| View previous topic :: View next topic |
| Author |
Message |
Caffeine
Guru
Joined: 17 Jul 2002
Posts: 401
Location: Melbourne, Australia
|
 Posted: Sat Sep 27, 2003 7:59 pm Post subject: |
 |
|
How about mysql replication? Add a few slave databases, put them behind an LVS. Of course, the sql in phpbb2 would need to be adjusted so that writes go to the master, and reads go to the LVS.
Or am I smoking crack? |
|
| Back to top |
|
 |
cerri
Bodhisattva
Joined: 05 Mar 2003
Posts: 2957
Location: # init S
|
| Posted: Sat Sep 27, 2003 8:30 pm Post subject: |
|
|
There is no need for LVS if you hack phpBB code.
_________________
Enjoy your freedom.
Sex is like hacking. You get in, you get out, and you hope you didnt leave something behind that can be traced back to you.
<----------------------->
Andrea Cerrito |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Sat Sep 27, 2003 9:00 pm Post subject: |
|
|
| cerri wrote: | | kurt, I saw your post: what has been done? |
the post that says "We'll be looking at integrating these into our site."? as in, future tense? 
nothing has been done yet.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
cerri
Bodhisattva
Joined: 05 Mar 2003
Posts: 2957
Location: # init S
|
| Posted: Sat Sep 27, 2003 11:32 pm Post subject: |
|
|
| klieber wrote: | | nothing has been done yet. |
mmmhhh... may I ask why?
I mean, I think it's worth involving phpbb guys to solve this issue: phpbb code isn't appearing as good as expected on this forum...
What about the architecture?
_________________
Enjoy your freedom.
Sex is like hacking. You get in, you get out, and you hope you didnt leave something behind that can be traced back to you.
<----------------------->
Andrea Cerrito |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Sat Sep 27, 2003 11:37 pm Post subject: |
|
|
| cerri wrote: | | mmmhhh... may I ask why? |
because I posted that question less than 12 hours ago, number 1. number 2, the guy hasn't responded with his code yet, so it's kind of hard for me to do much without it.
| cerri wrote: | | I mean, I think it's worth involving phpbb guys to solve this issue: phpbb code isn't appearing as good as expected on this forum... |
The phpbb guys are hard at work coding phpbb 2.2 which supposedly has much more efficient use of the database. As I've already mentioned, we cannot wait for them to make improvements in the code. We're at a point where we must do something now.
| cerri wrote: | | What about the architecture? |
what about it?
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
cerri
Bodhisattva
Joined: 05 Mar 2003
Posts: 2957
Location: # init S
|
| Posted: Sat Sep 27, 2003 11:45 pm Post subject: |
|
|
| cerri wrote: | I agree: the machine is powerful enough for db data you post, and MySQL 4 in recent tests were more efficient than Oracle. Anyway: have you tought to split the db on two machines (one for read and one for write)?
Anyway: in which way db and web servers communicate? What kind of machine serve forums.gentoo.org (www)? |
I meant this post. klieber, really thanks for your responses
I think this thread it's very interesting.
_________________
Enjoy your freedom.
Sex is like hacking. You get in, you get out, and you hope you didnt leave something behind that can be traced back to you.
<----------------------->
Andrea Cerrito
Last edited by cerri on Sat Sep 27, 2003 11:48 pm; edited 1 time in total |
|
| Back to top |
|
|
drakonite
l33t
Joined: 02 Nov 2002
Posts: 768
Location: Lincoln, NE
|
| Posted: Sat Sep 27, 2003 11:47 pm Post subject: |
|
|
What about defaulting to only search the past few months and a full search would have to be explicitly choosen? or does phpBB not support this?
_________________
Shoot Pixels Not People
My GPG/PGP Public key |
|
| Back to top |
|
|
crown
n00b
Joined: 15 Jun 2002
Posts: 64
|
| Posted: Sat Sep 27, 2003 11:57 pm Post subject: |
|
|
Being a Java person myself, I can't help but mention that PHP-based solutions just don't (necessarily) scale as well as Java-based ones. I was going to suggest using Jive Software's excellent forum package but it's not OSS.
I don't know of many open source Java forum solutions but some quick searching on freshmeat found mvnForum. It seems to offer many of the same features as phpBB but has a lot of the nice stuff that a Java solution gives you like database connetion pooling, that a PHP solution can't.
The problem with this particular package is that it hasn't yet reached 1.0. I've never seen it before so I can't speak for it's stability and certainly it hasn't matured yet, as phpBB and JiveForums have.
This isn't really an immediate solution to the problem, but I don't really see any other immediate solutions other than adding more hardware. |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Sun Sep 28, 2003 12:01 am Post subject: |
|
|
| cerri wrote: | I agree: the machine is powerful enough for db data you post, and MySQL 4 in recent tests were more efficient than Oracle. Anyway: have you tought to split the db on two machines (one for read and one for write)?
Anyway: in which way db and web servers communicate? What kind of machine serve forums.gentoo.org (www)? |
If we split the db onto two separate servers (hardware which we don't have, btw) I'm not sure that's going to solve the primary problem, which is people searching the forums and hitting the 1.5GB wordmatch table. We might gain some additional room to breathe, but I don't think it would solve all our problems.
As for the architecture, currently, the forums are split up with apache responsibilities being handled by a cluster of 3 dual xeon boxes and the DB being handled by a single dual xeon box as mentioned previously. The apache hardware is obviously quite sufficient and is not causing us any problems. It's also not our hardware, however (it belongs to OSU and we share it with some other sites) so we don't have the option of using those servers for other purposes (such as splitting up the DB)
The servers are all on the same local LAN, connected via gigabit ethernet.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Sun Sep 28, 2003 12:08 am Post subject: |
|
|
| drakonite wrote: | | What about defaulting to only search the past few months and a full search would have to be explicitly choosen? or does phpBB not support this? |
It is one option we're considering.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
cerri
Bodhisattva
Joined: 05 Mar 2003
Posts: 2957
Location: # init S
|
| Posted: Sun Sep 28, 2003 12:08 am Post subject: |
|
|
| klieber wrote: | | If we split the db onto two separate servers (hardware which we don't have, btw) I'm not sure that's going to solve the primary problem, which is people searching the forums and hitting the 1.5GB wordmatch table. We might gain some additional room to breathe, but I don't think it would solve all our problems. |
Well, we are discussing possible solutions
Anyway, the db splitting maybe interesting even for search: if hacking phpBB code is possible, probably it's possible to setup a db replica just to searching queries...
| klieber wrote: | | As for the architecture, currently, the forums are split up with apache responsibilities being handled by a cluster of 3 dual xeon boxes and the DB being handled by a single dual xeon box as mentioned previously. The apache hardware is obviously quite sufficient and is not causing us any problems. It's also not our hardware, however (it belongs to OSU and we share it with some other sites) so we don't have the option of using those servers for other purposes (such as splitting up the DB) |
.
Ok. I asked it because the "View posts from last 24 hours" is currently reporting a php error, not a db error, so I thought that maybe a php tuning is necessary (and can be done on virtualhost too).
| klieber wrote: | | The servers are all on the same local LAN, connected via gigabit ethernet. |
.
Mmmhhh... this let me think that there might be any other kind of problem... How such architecture can't be sufficient to serve a 2gb db? 
Finally: is installed any kind of php accelerator? is it possible to see any phpinfo page?
_________________
Enjoy your freedom.
Sex is like hacking. You get in, you get out, and you hope you didnt leave something behind that can be traced back to you.
<----------------------->
Andrea Cerrito |
|
| Back to top |
|
|
klieber
Bodhisattva
Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA
|
| Posted: Sun Sep 28, 2003 12:12 am Post subject: |
|
|
| crown wrote: | | Being a Java person myself, I can't help but mention that PHP-based solutions just don't (necessarily) scale as well as Java-based ones. I was going to suggest using Jive Software's excellent forum package but it's not OSS. |
If we do go to a commercial package, it's going to be vBulletin. It's proven to be scalable enough to meet our needs, both now and as we continue to grow.
Nothing against Java, mind you -- we just have quite a few more PHP experts on staff than Java ones, so PHP is better for our particular needs.
--kurt
_________________
The problem with political jokes is that they get elected |
|
| Back to top |
|
|
crown
n00b
Joined: 15 Jun 2002
Posts: 64
|
| Posted: Sun Sep 28, 2003 12:27 am Post subject: |
|
|
| klieber wrote: | | If we do go to a commercial package, it's going to be vBulletin. It's proven to be scalable enough to meet our needs, both now and as we continue to grow. |
I think the difference between those sites and the gentoo forums is that they are more discussion oriented. While they may have more posts and/or more members I doubt they see the same search volume as a support oriented site like forums.gentoo.org.
Also (you may have researched this or not) we don't know what kind of hardware those sites are running on and what they are using on the backend. |
|
| Back to top |
|
|
Westech
n00b
Joined: 11 Sep 2003
Posts: 12
|
| Posted: Sun Sep 28, 2003 2:03 am Post subject: |
|
|
One option to reduce the size of the word match file is to re-index it with a longer minimum word length and/or a shorter maximum word length. While this may reduce the usefulness of the search function, it's better than everything grinding to a halt.
Other options that will reduce the load on mysql (but not the search index size, so may not improve things much) are to disable avatars and remove nonessential things like poster's location from each post. Each thing you eliminate could reduce the load on mysql quite a bit.
For instance, just disabling avatars and removing "location" from being displayed with each post could save 50 mysql queries per pageview:
(assuming 25 posts displayed per page, thats 25 avatars + 25 user locations that have to be looked up.) |
|
| Back to top |
|
|
fca
Guru
Joined: 22 Feb 2003
Posts: 346
Location: Netherlands
|
| Posted: Sun Sep 28, 2003 11:47 am Post subject: |
|
|
Is xapian ( http://www.xapian.org/index.php ) an option?
Their Omega-search is being used in the largest forum I know (with over 10million posts). |
|
| Back to top |
|
|
Moled
l33t
Joined: 09 Jul 2003
Posts: 635
|
| Posted: Mon Sep 29, 2003 8:34 am Post subject: |
|
|
| stonent wrote: | | I'm curious if they are using the gzip ability of phpBB. |
https://forums.gentoo.org/statistics.php
| Quote: | | Gzip compression Disabled |
from what is said it is searches that are killing the forums?
I would disable searches where they are not really needed, ie significanly older posts where info is ususally outdated, and specific forums such as gentoo chat, off the wall etc etc |
|
| Back to top |
|
|
denis09
n00b
Joined: 22 Oct 2002
Posts: 14
|
| Posted: Mon Sep 29, 2003 3:59 pm Post subject: phpBB "search" vs real FreeTextIndexed searches.. |
|
|
I have had the same problems with phpBB like kurt mentions.. No not an equal sized forum, but far worse HW and more load from other parts on the same server.
I share Kurts frustrations with the phpBB search.. It does not scale at all. Firstly because a growing searchtable means slower posting, not only slower searching.. Secondly becuase the search is really bad at returning as many hits as possible, and at ranking them aswell.
I have some experience with searches, and have come to use and like a package called OpenFTS a lot.. It is a PostgreSQL solution however, but there is a way to use kind of the same searching in MySQL. It is called Fulltext Indexing.
The real solution to these problems would be to radically change phpBB, I am not sure this has been done in the 2.2 version or not, but it should be.. Instead of manually indexing words in a separate table, posts should be indexed with Fulltext Index type.. See Jeremy Zawodnys blogpost on the subject for more info : JZW Fulltext Blogentry
Alternatively one could as someone suggested use another product for the actual search.. In that case, the OpenFTS searchengine comes highly recommended from me.
phpBB will not be a viable alternative for a really serious, huuge board like this if the search is not ripped out and replaced.. The hardware you are currently running on Kurt should be more than enough to serve this forum ! |
|
| Back to top |
|
|
nephros
Advocate
Joined: 07 Feb 2003
Posts: 2139
Location: Graz, Austria (Europe - no kangaroos.)
|
| Posted: Wed Oct 01, 2003 7:28 pm Post subject: |
|
|
| pjp wrote: |
We are considering pruning specific forums (such as OTW). No idea what removing it from the "search all forums" option would do, but I think it should be searchable by selecting it specifically.
|
Hmmm, just a thought: would it help if searches were only possible per Forum Topic? I am not at all familiar with how phpbb handles searches, but if there were a seperate btree for each Forum that would mean about 10% the stress per search.
So users would probably have to search more than once, i.e once in Hardware and once in Multimedia if their sound isn't working, but the current way it wouid search networking too, which is completely bogus and only ballast on the search.
Would that work?
[EDIT:] just realized that a similar question has been asked above. Sorry for duplicating, i didn't see the second page . 
_________________
Please put [SOLVED] in your topic if you are a moron. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Wed Oct 01, 2003 8:12 pm Post subject: |
|
|
| nephros wrote: | | So users would probably have to search more than once, i.e once in Hardware and once in Multimedia if their sound isn't working |
By the number of duplicate posts, alot of people don't search now, or for very long. Individual forum search would probably end up being the same as disabling search.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
nephros
Advocate
Joined: 07 Feb 2003
Posts: 2139
Location: Graz, Austria (Europe - no kangaroos.)
|
| Posted: Wed Oct 01, 2003 11:52 pm Post subject: |
|
|
| pjp wrote: | | By the number of duplicate posts, alot of people don't search now, or for very long. Individual forum search would probably end up being the same as disabling search. |
While I don't have the numbers, I don't think that is a valid argument: People not searching at all is a different problem altogether.
The problem at hand is providing the possibiltiy of search to those who do want to, while keeping the Forum useable for everybody. And the usability/efficiency of both the forum itself *and* the search function would be increased by individual searches would it not?
_________________
Please put [SOLVED] in your topic if you are a moron. |
|
| Back to top |
|
|
|
Gentoo Forums Feedback
