Linux Performance Tuning Questions

[njcwotx]

Greets everyone! I have some linux performance tuning questions. This is not Gentoo, but a Red Hat EL5 system, but I actually value the posters opinions here more. So, I posted in unsupported software to generate a little discussion on some topics that have come up recently with some of the output with top, vmstat, iostat and mpstat.

The server runs a front end application and database progress/QAD is the software. 8CPU's and 6GB Ram on an HP DL580 G3.

For some time my dba's and app guys have been saying users are reporting slowness and immediately ask for more RAM and CPU's without any evidence. I look at our stats and dont see the need. Here is my output. I want to know if there is anything wrong here or if I am missing something. I have particular interest in the wa (waiting on I/O) value as that looks of interest to me. Often I can see values but wonder what is considered out of normal parameters.

some specific starter questions:
1. How much disk cache is really useful/needed? I have seen anywhere from 2.5Gb-5Gb Disk Cache. As I have read and researched, free RAM in Linux is always low since Linux does not want to have unused wasted RAM and so turns it into disk cache. If this is so, does 2.5Gb of Disk Cache lead me to believe this is not a RAM issue?

2. How high of a iowait time is too high? Sometimes I see single digits, others I see double digits in "wa" using the vmstat command or "%iowait" using mpstat. I am wondering if my problem may be one of Disk I/O load than RAM.

3. What is load average mean in Top?

[micmac]

Which IO scheduler is used (if your kernel supports that)? Maybe try deadline if available, it's supposed to do a decent jobs when using databases. Also, is the system 32 bit or 64 bit? 64 bit should make a difference with DBs and a big box like yours.

[njcwotx]

its a 32bit system. I am not sure what you mean by which ioscheduler, how can I tell and what options are there?

I also have another new question...when I run top, I have some users max cpu at 100% but the Total CPU is 10% or something. Is there a way to divide out individual cpu usage values? Maybe the threads are not getting balanced across CPU's well enough, perhaps this is where the ioscheduler comes into play.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

[njcwotx]

Ok, your question got me to look this up and I learned something new. Thanks!

I am using the cfq scheduler, but deadline is an option, in fact noop and anticipatory are too. I will keep this out as an option to try if we get to tweaking.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

[Neo2]

I don't know how fedora kernel packages work, anyway, seems like your problem is related to non-optimized kernel. Usually binary distros build up generic kernels that include of all the possible hardware that the user might need, hence big kernel images and potentially slow execution (not to mention possible hardware conflicts). If I were you I'd dig for some easy way to compile your own kernel on that system.
You would then manually tune things like the disk scheduler, the specific CPU model your kernel is built for, which memory allocator to use and which drivers you really need to include for your system to work properly. You could install the kernel sources with your package manager and then use the gentoo linux kernel guide as a reference in choosing the right options. This will result in a lighter, faster and tuned kernel.
That's just a suggestion. It should provide the performance boosts you're asking for
_________________
Neo2
Unofficial minimal liveCD for x86/amd64 w/reiser4+truecrypt

[micmac]

You can change the ioscheduler on the fly, no need to reboot to test that. "deadline" should be good for what your box is doing.

[njcwotx]

Thanks.

Any ideas on how much disk cache is optimal? Does it do any good to get mega gigs of disk cache or is it a simple way to lower your cost/benefit ratio?
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

[NeddySeagoon]

njcwotx,

An 8 way 64 bit capable box with 6GB of RAM and a 32 bit bit install.

You can only address 4Gb of RAM directly with a 32 bit install, the rest is paged like expanded (not extended) RAM in days gone by.
This uses the PAE to do the best it can. That may explain the large amount of free RAM and 70MB used in swap.
You are running out of space with the per process 3G/1G memory split.

I suspect you need to move to a 64 bit OS so you can get back to a linear address space. This won't help if your application is only available in 32 bit form.
You would get to use the top half of all your cores too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[SeaTiger]

njcwotx:

Is the "free", "top", "iostat" and "mpstat" taken during the time user complain or after?

It will be more useful to have a "snapshot" when user feel the "slowness".

I am seeing 282 users, are they all (or 'many') using _progress ?? If yes, maybe memory is an issue. My guess is the 70M swap was created during the peek load (and the swap maybe larger than 70M during that time), which is long before you run "top".

Regarding swap memory, according to my experience, whenever the machine start to use swap, it is going to be slow. Basically, you want to minimize swap usage.

Can you check if your kernel support PAE?
Try the following:

[njcwotx]

The stats above are done from a few days later, but I did check during the 'slowness'. At the time I saw no paging, and that 70M of swap was there all week. Right now its 17M (there was a reboot saturday) , I think when your swap file is low and not regularly accessed a small amount of swap should not be an issue. Im fairly certain we have no paging at this time.

However, I have been really watching the cached value. It fluctuates quite a bit, I have seen it go from almost 5Gb of RAM to 2.5Gb of ram over the course of a day, then the next morning after everyone logged out and went home the numbers go back up. It sounds reasonable. As far as the slowness, I dont think its an all-day-long affair. I think its intermittent and possibly tied to when someone does a large report. I am waiting for a re-occurance to re-snap the values. I am predicting some kind of disk I/O bottleneck and will be looking at the 'waiting on I/O' values and iostats as well as the available cache. I am wondering if my cached RAM is fairly low at the time it gets slow.

[SeaTiger]