View previous topic :: View next topic |
Author |
Message |
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Sat Jan 17, 2004 9:34 am Post subject: |
|
|
aengus9,
did the ext3 patch helped you at all (you are using ext3, right)?
Your first post in this thread said that your system was thrashed after updatedb run, is this still happening, or is the only annoyance that memory is not reclaimed?
As for as reclaiming, read my next post.
BTW, when reading free output, the most relevant numbers are on the second line "-/+ buffers/cache", these should show real used and available memory.
It should be just sum of free + buffers and caches.
Oh, and do not reboot until necessary. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Sat Jan 17, 2004 10:06 am Post subject: |
|
|
Everybody following this thread
The bug was officially closed as resolved.
The ext3 problem was fixed with the patch that made it into gentoo 2.4.22-r5 kernel.
As for as not reclaimng memory, the reply from developer to my free command output was that it is not bug and that it happens on every system he has seen. I did search on this problem and quite a lot of people noticed this problems over past two years, it even appeared several times on kernel mailing list leading to technical discussions (involving Linus) about reclaimng memory saying basically that memory is in fact available but is reclaimed only when there is no other memory avaialable -at least that is my impression. A couple of people (not on kernel mailing list) used a C program which allocates lots of memory (using malloc) to confirm that this memory can be in fact used by other applications.
So, it seems like this issue is not a bug, but a kernel memory management feature that we do not understand (or perhaps free command does not handle report memory usage correclty in some special situations). Also, if this would be a real memory leak this would lead to crashes, which nobody reported.
One last thing: Strangely, it seems to me that this does not happen on redhat 9 box I use at work. I will try to test it on Monday. |
|
Back to top |
|
|
aengus9 n00b
Joined: 21 May 2002 Posts: 25
|
Posted: Sat Jan 17, 2004 7:51 pm Post subject: |
|
|
I'm sorry, I should have been more clear. Yes, the patch did take care of my problem. Before the patch, my system would be completely thrashed after running updatedb. After applying the patch, while I can't see memory being freed, my system does seem to function well enough now.
Thanks for the explanation mhodak, I appreciate the englightenment. I'll hold off on the scheduled reboots. I do enough of that with my windows machines anyway. |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 1:49 am Post subject: |
|
|
mhodak wrote: | Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.
Mine: 2.4.22-gentoo-r2, ReiserFS |
I came here researching exactly the same problem. I have experienced it on all of the 2.4.22 kernels. I am presently using 2.4.22-gentoo-r4 with reiserfs and several mounted nfs directories. I believe locatedb is ignoring the nfs directories. (hope so anyway).
I run 1gb ram, and my memory jumps to 580mb used and stays there after running updatedb. Normally light usage stays about 280mb, and any increases are freed immediately upon exit of the extra programs.
I have the kernel set for 4gb high memory configuration and tested both with and without the 'himem' checkbox below that. no change.
I do not recall this problem when I was running 2.4.20 but then I also had a different hardware resource configuration and at that time was running vmware consistantly which inflated my swap to the point I had to reboot every 2 days.
Chuck |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 2:15 am Post subject: |
|
|
mhodak wrote: | Everybody following this thread
The bug was officially closed as resolved.
The ext3 problem was fixed with the patch that made it into gentoo 2.4.22-r5 kernel.
As for as not reclaimng memory, the reply from developer to my free command output was that it is not bug and that it happens on every system he has seen. I did search on this problem and quite a lot of people noticed this problems over past two years, it even appeared several times on kernel mailing list leading to technical discussions (involving Linus) about reclaimng memory saying basically that memory is in fact available but is reclaimed only when there is no other memory avaialable -at least that is my impression. A couple of people (not on kernel mailing list) used a C program which allocates lots of memory (using malloc) to confirm that this memory can be in fact used by other applications.
So, it seems like this issue is not a bug, but a kernel memory management feature that we do not understand (or perhaps free command does not handle report memory usage correclty in some special situations). Also, if this would be a real memory leak this would lead to crashes, which nobody reported.
One last thing: Strangely, it seems to me that this does not happen on redhat 9 box I use at work. I will try to test it on Monday. |
I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!
Sorry, but there is something odd about the 2.4.22+ kernels. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Tue Jan 20, 2004 3:42 am Post subject: |
|
|
RioFL wrote: |
I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!
Sorry, but there is something odd about the 2.4.22+ kernels. |
This thing bothers me as well, I do not like updatedb caching and not releasing memory properly. As I wrote above, it can be reclaimed if really needed , but it is 50% of my memory and I really do not care about updatedb running fast since it runs over night anyway. At least the second run of updatedb do not take any more memory.
Perhaps the best solution would be to open a new bug and see if we can get developers to do something about it, since even 2.6 kernels experience this behavior. I will do this if my test on RH box will show that RH kernel works as expected. |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 6:55 am Post subject: |
|
|
mhodak wrote: | RioFL wrote: |
I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!
Sorry, but there is something odd about the 2.4.22+ kernels. |
This thing bothers me as well, I do not like updatedb caching and not releasing memory properly. As I wrote above, it can be reclaimed if really needed , but it is 50% of my memory and I really do not care about updatedb running fast since it runs over night anyway. At least the second run of updatedb do not take any more memory.
Perhaps the best solution would be to open a new bug and see if we can get developers to do something about it, since even 2.6 kernels experience this behavior. I will do this if my test on RH box will show that RH kernel works as expected. |
I am beginning to suspect maybe more than one thing. I downgraded back to 2.4.20-gentoo-r7 after 4 hours of playing with aa-sources, and gentoo sources and various combinations of configurations with and without himem support.
This is really confusing. All kernels left memory high as reported by gkrellm2 and free. Except 2.4.20. That one used 890mb with the kernel configured for no highmem support at all so I lose 100mb or so of top ram. It has been an hour since I ran it. Free still reports 886mb used, however gkrellm2 reports 304mb used which I expect considering with highmem support my usual is 240mb.
This machine I have this trouble on has the latest of everything on it. The other machine which also uses the 2.4.20-r7 kernel, has not been updated in a while. That one also only has 512mb ram. Behavior on that machine is perfect. Memory goes up to close to 450mb, and as soon as updatedb is finished, it drops right back down to 145mb where it normally is. What appeared to be a simple case of finding which version kernel this began with, has suddenly turned into a possible shared library/gcc/whoknowswhat nightmare. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Tue Jan 20, 2004 11:13 am Post subject: |
|
|
I also think that when using 2.4.20 kernel my memory usage was lower, but I do not have this kernel anymore. I am ruuning gkrellm and to my recollection the used memory bar was showing about half, but now it is close to full.
BTW, in my experience, gkrellm and free alway gave the same (within 5 MB or so). |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 12:55 pm Post subject: |
|
|
mhodak wrote: | I also think that when using 2.4.20 kernel my memory usage was lower, but I do not have this kernel anymore. I am ruuning gkrellm and to my recollection the used memory bar was showing about half, but now it is close to full.
BTW, in my experience, gkrellm and free alway gave the same (within 5 MB or so). |
I don't have the ebuild stuff any more I just tarred up my directory from the other machine. If you want it I can put it on a site you can download it. It is 200mb because I did not clear out the object modules since I didnt want to disturb the config for that machine. The free/gkrellm agreement has always been my experience too. but this time around it is different. Currently my gkrellm is telling me I have 288mb used yet my free command is way up. here it is:
total used free shared buffers cached
Mem: 902584 894704 7880 0 350996 249268
-/+ buffers/cache: 294440 608144
Swap: 498004 6968 491036
VERY odd behavior.
Chuck |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Tue Jan 20, 2004 9:39 pm Post subject: |
|
|
I just did tests on RH machine which uses 2.4.20 kernel and here are the results (the machine is freshly rebooted):
Code: |
# free -m
total used free shared buffers cached
Mem: 249 43 206 0 5 20
-/+ buffers/cache: 16 233
Swap: 517 0 517
#updatedb
# free -m
total used free shared buffers cached
Mem: 249 229 20 0 64 22
-/+ buffers/cache: 142 107
Swap: 517 0 517
|
Thus it eats about half of the memory, the same as 2.4.22 kernel on my gentoo machine. But doing anything intensive (installing rpm for new kernel, running nvidia-installer) frees up most of the memory. Here is memory after "nvidia-installer -f"
Code: |
#free -m
total used free shared buffers cached
Mem: 249 210 38 0 64 107
-/+ buffers/cache: 38 210
Swap: 517 0 517
|
I have also tried rerunning of upgradedb at this point to see if it wants some more memory, but memory usage stays the same after the rerun.
It seems as if the memory held by updatedb can in fact be easily freed when needed in 2.4.20, but I still do not undertsand why it is not freed immediately as it is the case for other applications (or perhaps free is wrong?). The thing is that with 2.4.22 kernel on my Gentoo box, doing emerge or something similar does not seem to free up the memory, but it may be because the gentoo box has twice the memory of RH box.
I realized that 2.4.20 kernel is still available through portage so I will try it
later today and test if its memory management is any better.
In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)? |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 9:54 pm Post subject: |
|
|
mhodak wrote: | <snipped for brevity>
It seems as if the memory held by updatedb can in fact be easily freed when needed in 2.4.20, but I still do not undertsand why it is not freed immediately as it is the case for other applications (or perhaps free is wrong?). The thing is that with 2.4.22 kernel on my Gentoo box, doing emerge or something similar does not seem to free up the memory, but it may be because the gentoo box has twice the memory of RH box.
I realized that 2.4.20 kernel is still available through portage so I will try it
later today and test if its memory management is any better.
In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)? |
I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.
btw, I found that acpi and 2.4.20 and highmem and nvidia to not co-exist well at all. I am running the kernel without highmem, using the default 3gb split, and without acpi. It appears the acpi code in that kernel is a bit strange. I lose the use of about 100mb ram that way, but everything appars to be more than good, and even though free still is reporting 890mb used after 15 hrs, gkrellm is reporting presently 310mb used which would be very close to my expectations for what I have running at the moment.
Chuck |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Tue Jan 20, 2004 10:10 pm Post subject: |
|
|
RioFL wrote: | mhodak wrote: |
In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)? |
I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.
Chuck |
Which kernel is this? Did you see (in free or gkrellm) that the amount of used memory decreased when application was done? |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Tue Jan 20, 2004 10:13 pm Post subject: |
|
|
mhodak wrote: | RioFL wrote: | mhodak wrote: |
In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)? |
I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.
Chuck |
Which kernel is this? Did you see (in free or gkrellm) that the amount of used memory decreased when application was done? |
everything but updatedb reduces memory immediately upon exit. this is kernel 2.4.20-gentoo-r7
I went back to that after running 2.4.22 and having this memory problem. |
|
Back to top |
|
|
___c___ n00b
Joined: 04 Dec 2003 Posts: 61
|
Posted: Wed Jan 21, 2004 1:01 am Post subject: |
|
|
Code: |
$ uname -a
Linux GEN2 2.4.22-gentoo-r5 #1 Thu Jan 15 21:30:08 CET 2004 i686 AMD Athlon(tm) processor AuthenticAMD GNU/Linux
$ grep "hda3" /etc/fstab
/dev/hda3 / reiserfs noatime 0 0
$ free -m
total used free shared buffers cached
Mem: 756 108 647 0 6 47
-/+ buffers/cache: 55 700
Swap: 517 0 517
$ updatedb
$ free -m
total used free shared buffers cached
Mem: 756 714 41 0 326 51
-/+ buffers/cache: 335 420
Swap: 517 0 517
$ ooffice
$ free -m
total used free shared buffers cached
Mem: 756 735 20 0 288 108
-/+ buffers/cache: 338 417
Swap: 517 0 517
|
|
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Wed Jan 21, 2004 1:11 am Post subject: |
|
|
___c___ wrote: | Code: |
$ uname -a
Linux GEN2 2.4.22-gentoo-r5 #1 Thu Jan 15 21:30:08 CET 2004 i686 AMD Athlon(tm) processor AuthenticAMD GNU/Linux
$ grep "hda3" /etc/fstab
/dev/hda3 / reiserfs noatime 0 0
$ free -m
total used free shared buffers cached
Mem: 756 108 647 0 6 47
-/+ buffers/cache: 55 700
Swap: 517 0 517
$ updatedb
$ free -m
total used free shared buffers cached
Mem: 756 714 41 0 326 51
-/+ buffers/cache: 335 420
Swap: 517 0 517
$ ooffice
$ free -m
total used free shared buffers cached
Mem: 756 735 20 0 288 108
-/+ buffers/cache: 338 417
Swap: 517 0 517
|
|
presently my gkrellm is at 326mb but i have a lot going on. however free has never changed from last night, at least not any noticable amount:
gndmstr@eron gndmstr $ free -m
total used free shared buffers cached
Mem: 881 874 7 0 80 465
-/+ buffers/cache: 327 553
Swap: 486 11 475
i have 1024mb ram but with himem turned off 881 (900) is what the kernel sees. as you can see, free still sees 874mb used, but gkrellm reports 326 which is more in keeping with what i am doing. |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Wed Jan 21, 2004 1:12 am Post subject: |
|
|
Oh, I don't think I ever mentioned, I am running reiserfs on everything but /boot which is ext2
Chuck |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Wed Jan 21, 2004 4:34 am Post subject: |
|
|
Guys,
I installed 2.4.20 kernel on my Gentoo box and compared it to 2.4.22 kernel. 2.4.20 works better with respect to the updatedb problem. First running updatedb eats less memory (but it still does not release it properly) but when trying after running emerge most of the memory is released. For 2.4.22 kernel running emerge worsens the situation.
I am about to fill a bug report, do not know what else to do. |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
|
Back to top |
|
|
ekoontz n00b
Joined: 18 Apr 2002 Posts: 67 Location: San Francisco, California
|
Posted: Sat Jan 24, 2004 9:06 am Post subject: |
|
|
I had this problem also; not knowing what else to do, I tried 2.6.1-rc1, and it seems to have gone away.
I know that doesn't help much with finding out the root cause, but I guess it qualifies as a workaround! _________________ In Soviet Gentoo, portage emerges -u! |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Sat Jan 24, 2004 11:55 am Post subject: |
|
|
Great to hear that there is a kernel in which this weird memory behavior was fixed !!! |
|
Back to top |
|
|
tomaw Guru
Joined: 26 Mar 2003 Posts: 429 Location: UK
|
Posted: Sat Jan 24, 2004 11:57 am Post subject: |
|
|
I seem to still suffer from the memory reporting being wrong in 2.6.1 final - hoping to try 2.6.2 within a few days and see if that has gone anywhere to fixing it... _________________ Tom Wesley |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Sat Jan 24, 2004 12:01 pm Post subject: |
|
|
ekoontz wrote: | I had this problem also; not knowing what else to do, I tried 2.6.1-rc1, and it seems to have gone away.
I know that doesn't help much with finding out the root cause, but I guess it qualifies as a workaround! |
never played with the dev sources before because I always thought that they were supposed to be too unstable... i see bugs images on 2.6.1-r1, don't find an rc1, but there is a 2.6.1 that looks like a 'release' so I am going to try that one. 2.4.20 is just too 'backward' for me. I notice it is slightly slower than 2.4.22, video isn't as crisp and of course memory management is worse with buggy highmem code.. so I am going to try the 2.6.1 this morning and see what it does.
My biggest concern is if I switch to this, can I still go backward to lower versions? I heard somewhere that 2.6.1 did away with devfs which is the core of my system.
Chuck |
|
Back to top |
|
|
ekoontz n00b
Joined: 18 Apr 2002 Posts: 67 Location: San Francisco, California
|
Posted: Sat Jan 24, 2004 5:16 pm Post subject: |
|
|
RioFL, devfs seems to be considered deprecated in 2.6, but still you can choose it (as I did), and it seems to work fine.
You can still boot older kernels; devfs doesn't change anything permanently about the /dev directory as far as I Know. (it just creates /dev entries dynamically which go away at the next boot).
tomaw, here's some data; i've been up for 4 days so updatedb has been running every night through cron..
(the load is high because I'm emerging a bunch of stuff )
Code: |
hiro-tan(ekoontz)[~] $ uptime
09:15:25 up 4 days, 8:11, 2 users, load average: 1.01, 1.07, 1.06
hiro-tan(ekoontz)[~] $ free -m
total used free shared buffers cached
Mem: 216 203 12 0 21 85
-/+ buffers/cache: 96 120
Swap: 1027 60 967
hiro-tan(ekoontz)[~] $ uname -a
Linux hiro-tan.org 2.6.1-rc1 #1 SMP Fri Jan 16 00:04:54 PST 2004 i686 VIA Samuel 2 CentaurHauls GNU/Linux
|
_________________ In Soviet Gentoo, portage emerges -u! |
|
Back to top |
|
|
RioFL Guru
Joined: 31 Oct 2002 Posts: 407
|
Posted: Sat Jan 24, 2004 6:25 pm Post subject: |
|
|
ekoontz wrote: | RioFL, devfs seems to be considered deprecated in 2.6, but still you can choose it (as I did), and it seems to work fine.
You can still boot older kernels; devfs doesn't change anything permanently about the /dev directory as far as I Know. (it just creates /dev entries dynamically which go away at the next boot).
tomaw, here's some data; i've been up for 4 days so updatedb has been running every night through cron..
(the load is high because I'm emerging a bunch of stuff )
Code: |
hiro-tan(ekoontz)[~] $ uptime
09:15:25 up 4 days, 8:11, 2 users, load average: 1.01, 1.07, 1.06
hiro-tan(ekoontz)[~] $ free -m
total used free shared buffers cached
Mem: 216 203 12 0 21 85
-/+ buffers/cache: 96 120
Swap: 1027 60 967
hiro-tan(ekoontz)[~] $ uname -a
Linux hiro-tan.org 2.6.1-rc1 #1 SMP Fri Jan 16 00:04:54 PST 2004 i686 VIA Samuel 2 CentaurHauls GNU/Linux
|
|
I put it in and am not using devfs. Decided to get into the spirit of 2.6.1 and 'go with the flow'. Everything appears to work just fine except my sensors and my logitech quickcam module which won't compile now. They eliminated LM80 sensors although the VIAPro stuff is in there, gkrellm isnt seeing the sensors at all:( will have to research it. Outside of that, my video is better, everything seems quicker.
Unfortunately the updatedb problem is still there so for now I have decided to eliminiate it from the nightly run and I will run it manually then reboot right after.. maybe a few times a week.
Only other problem I see which isnt really a problem, is the boot script warns me devfs is not installed, but it does not appear to affect anything.
Chuck |
|
Back to top |
|
|
mhodak Veteran
Joined: 15 Nov 2003 Posts: 1218
|
Posted: Thu Jan 29, 2004 11:48 am Post subject: |
|
|
Guys,
I checked Kernel mailing list to see if there were any replies to the post with our problem, but there are none, which means that there is no solutiom or explanation for the problem we are experiencing. Considering that this exists even in 2.6 kernels I would really like to see an explanation from somebody who knows about kernel memory management.
I am thinking about posting there our problem again, it may get noticed.
On the other hand, my machine has 10 days uptime now, I do updatedb everyday and besides having almost 100% used memory (reported by gkrellm), it does not seem to affect me. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|