View previous topic :: View next topic |
Author |
Message |
HELPcho n00b
Joined: 10 Jun 2006 Posts: 9 Location: Plovdiv
|
Posted: Tue Aug 25, 2009 8:36 am Post subject: too many "buffers" in kernel |
|
|
Hello,
I have a strange situation with on of my servers. I have two servers for image serving with let say 10 milion images and two servers with 32G ram each. First to say what is expected, buffers about 3G, and cached about 23-25G. The software was the same on the machines like glibc 2.6.1 and gcc 4.1.2, the hardware also 3ware controller and 2xcpus xeons. Serving round robin the content with nginx. So the first server has 3G buffers, and 25G cached and is cool, no problem with I/O, the second server is IO starving and the RAM is 14G "buffers" and 15G "cached". Apparently the second server needs more ram for "cached", but I cannot figure out why this is happening, why the first fills up to 3G "buffers" and everything else goes to cached, but the second fills above 4 times more "buffers". The only great difference is the kernel version - the first is with old 2.6.22.18 kernel, and the second is 2.6.29.1 (the same with 2.6.30).
First server:
total used free shared buffers cached
Mem: 32964644 32806120 158524 0 3001732 23770288
~ # cat /proc/sys/fs/inode-state
4971217 852167 0 0 0 0 0
~ # cat /proc/sys/fs/dentry-state
4215476 4193181 45 0 0 0
Second server:
total used free shared buffers cached
Mem: 32944792 21438124 11506668 0 8205736 10484772
~ # cat /proc/sys/fs/inode-state
1659561 42 0 0 0 0 0
~ # cat /proc/sys/fs/dentry-state
1736219 1718366 45 0 0 0
At the moment the socond server is still caching, because the I used echo 3 > drop_caches to clear all the cache, but when the RAM fills up it the ratio for the buffers/cached will be 14G/15G or 14/16G. |
|
Back to top |
|
|
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Tue Aug 25, 2009 11:41 am Post subject: |
|
|
The only time I've seen a lot of 'buffers' being used, is when doing raw I/O to a device, without going through a filesystem. Is there something on the second server doing this kind of raw I/O? |
|
Back to top |
|
|
HELPcho n00b
Joined: 10 Jun 2006 Posts: 9 Location: Plovdiv
|
Posted: Tue Aug 25, 2009 6:12 pm Post subject: |
|
|
No, on the server is running just nginx and nfs, there is no special device. Just 3ware controller in RAID10 with stripe size 16K. Ammm, I checked and the stripe size to the raid10 on the first server is 64K, but I don't think this could cause the strange behavour. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|