Apache hangs the machine with iowait - config help?

Message

Ravenium · Post by **Ravenium** » Wed Sep 08, 2004 2:05 pm

Hello again,

I recently helped in setting up an ad content server for a gaming website. It's a fairly busy server (I have to prune the log about once a day or it fills to 2 gig, so it gets hit a bit). Luckily, the box is a hyperthreaded dual 2.8 xeon with 2 gigs of ram.

However, about once every few days, the box pretty much slows to a crawl/lock. All processors are shown with 1-2% active cpu, 70-90% iowait. Now, I'm positive that this is disk bound, and upon observations I saw that nearly all of the 2 gigs of swap was in use. Most of the resources seem to be used by httpd (apache 2.0.49), nearly 200-500 threads. Each proc/thread only takes 5-10 meg of mem/swap, but it of course adds up.

I have a distinct suspicion that one of my co-sysadmins has been toying with the apache config in a well-meaning effort to "speed up" the box, and it's having counterproductive results.

My question is this: What should I look for and/or be doing in terms of apache config to remedy this issue? Even if the ad content is slower, I'd rather have it as such than lock the box.

Any suggestions would be greatly appreciated!

-Barry

adaptr · Post by **adaptr** » Wed Sep 08, 2004 2:45 pm

Step 1: LART the person responsible - hard.
Step 2: stop apache
Step 3: copy the apache config
Step 4: re-instate the original (outofthebox) config and run it.

Now you can start comparing the MPM settings between the original config and the fecked up one.

Standard prefork settings are 5-10 servers, 150 threads per server, unlimited client requests per thread.
For a high-load server it may be wise to configure this, so each thread gets killed after, say, 10K requests.

That should stop those processes from running away with resources.

5MB of unique memory per process is way to much.

Ravenium · Post by **Ravenium** » Wed Sep 08, 2004 7:34 pm

Well, so far, so good. No runaway Apache. I've pretty much reverted back to vanilla settings, minus setting threads to die after 10k requests (when in doubt, I figure try the proposed suggestion first!).

Now while responses are up, response time is rather sluggish. Apache's grabbing about 40-60% of active cpu (thankfully very little of this is iowait).

Is it a sign of the apocalypse, or have I missed something that I might end up LARTing myself over? (naaah...when in doubt, blame those who touched it before you)

-Barry

adaptr · Post by **adaptr** » Thu Sep 09, 2004 12:01 am

Well, you can run the apachebench program included with apache (called ab2 in apache 2) and simulate runs with exactly the conditions you want.

I'd strongly recommend you do that to get a "feel" for what exactly are the peak conditions for this particular server.
If you get any useful numbers out of it you can then either adjust or upgrade to suit.