Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
2.4.22-gentoo-r2 eats memory?!
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Sat Jan 17, 2004 9:34 am    Post subject: Reply with quote

aengus9,
did the ext3 patch helped you at all (you are using ext3, right)?
Your first post in this thread said that your system was thrashed after updatedb run, is this still happening, or is the only annoyance that memory is not reclaimed?
As for as reclaiming, read my next post.

BTW, when reading free output, the most relevant numbers are on the second line "-/+ buffers/cache", these should show real used and available memory.
It should be just sum of free + buffers and caches.

Oh, and do not reboot until necessary.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Sat Jan 17, 2004 10:06 am    Post subject: Reply with quote

Everybody following this thread
The bug was officially closed as resolved.
The ext3 problem was fixed with the patch that made it into gentoo 2.4.22-r5 kernel.

As for as not reclaimng memory, the reply from developer to my free command output was that it is not bug and that it happens on every system he has seen. I did search on this problem and quite a lot of people noticed this problems over past two years, it even appeared several times on kernel mailing list leading to technical discussions (involving Linus) about reclaimng memory saying basically that memory is in fact available but is reclaimed only when there is no other memory avaialable -at least that is my impression. A couple of people (not on kernel mailing list) used a C program which allocates lots of memory (using malloc) to confirm that this memory can be in fact used by other applications.

So, it seems like this issue is not a bug, but a kernel memory management feature that we do not understand (or perhaps free command does not handle report memory usage correclty in some special situations). Also, if this would be a real memory leak this would lead to crashes, which nobody reported.

One last thing: Strangely, it seems to me that this does not happen on redhat 9 box I use at work. I will try to test it on Monday.
Back to top
View user's profile Send private message
aengus9
n00b
n00b


Joined: 21 May 2002
Posts: 25

PostPosted: Sat Jan 17, 2004 7:51 pm    Post subject: Reply with quote

I'm sorry, I should have been more clear. Yes, the patch did take care of my problem. Before the patch, my system would be completely thrashed after running updatedb. After applying the patch, while I can't see memory being freed, my system does seem to function well enough now.

Thanks for the explanation mhodak, I appreciate the englightenment. I'll hold off on the scheduled reboots. I do enough of that with my windows machines anyway.
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 1:49 am    Post subject: Reply with quote

mhodak wrote:
Whoever comes here and experience this problem, can you post your kernel version and filesystem?
May help to identify the bug.

Mine: 2.4.22-gentoo-r2, ReiserFS


I came here researching exactly the same problem. I have experienced it on all of the 2.4.22 kernels. I am presently using 2.4.22-gentoo-r4 with reiserfs and several mounted nfs directories. I believe locatedb is ignoring the nfs directories. (hope so anyway).

I run 1gb ram, and my memory jumps to 580mb used and stays there after running updatedb. Normally light usage stays about 280mb, and any increases are freed immediately upon exit of the extra programs.

I have the kernel set for 4gb high memory configuration and tested both with and without the 'himem' checkbox below that. no change.

I do not recall this problem when I was running 2.4.20 but then I also had a different hardware resource configuration and at that time was running vmware consistantly which inflated my swap to the point I had to reboot every 2 days.

Chuck
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 2:15 am    Post subject: Reply with quote

mhodak wrote:
Everybody following this thread
The bug was officially closed as resolved.
The ext3 problem was fixed with the patch that made it into gentoo 2.4.22-r5 kernel.

As for as not reclaimng memory, the reply from developer to my free command output was that it is not bug and that it happens on every system he has seen. I did search on this problem and quite a lot of people noticed this problems over past two years, it even appeared several times on kernel mailing list leading to technical discussions (involving Linus) about reclaimng memory saying basically that memory is in fact available but is reclaimed only when there is no other memory avaialable -at least that is my impression. A couple of people (not on kernel mailing list) used a C program which allocates lots of memory (using malloc) to confirm that this memory can be in fact used by other applications.

So, it seems like this issue is not a bug, but a kernel memory management feature that we do not understand (or perhaps free command does not handle report memory usage correclty in some special situations). Also, if this would be a real memory leak this would lead to crashes, which nobody reported.

One last thing: Strangely, it seems to me that this does not happen on redhat 9 box I use at work. I will try to test it on Monday.


I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!

Sorry, but there is something odd about the 2.4.22+ kernels.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Tue Jan 20, 2004 3:42 am    Post subject: Reply with quote

RioFL wrote:


I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!

Sorry, but there is something odd about the 2.4.22+ kernels.


This thing bothers me as well, I do not like updatedb caching and not releasing memory properly. As I wrote above, it can be reclaimed if really needed , but it is 50% of my memory and I really do not care about updatedb running fast since it runs over night anyway. At least the second run of updatedb do not take any more memory.

Perhaps the best solution would be to open a new bug and see if we can get developers to do something about it, since even 2.6 kernels experience this behavior. I will do this if my test on RH box will show that RH kernel works as expected.
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 6:55 am    Post subject: Reply with quote

mhodak wrote:
RioFL wrote:


I don't buy their excuse. For this simple reason that when I went back to 2.4.20-gentoo-r7 kernel [in the past 15 min] my locatedb memory usage NEVER exceeded 290mb total and freed itself (100% of it) immediately upon exit!!

Sorry, but there is something odd about the 2.4.22+ kernels.


This thing bothers me as well, I do not like updatedb caching and not releasing memory properly. As I wrote above, it can be reclaimed if really needed , but it is 50% of my memory and I really do not care about updatedb running fast since it runs over night anyway. At least the second run of updatedb do not take any more memory.

Perhaps the best solution would be to open a new bug and see if we can get developers to do something about it, since even 2.6 kernels experience this behavior. I will do this if my test on RH box will show that RH kernel works as expected.



I am beginning to suspect maybe more than one thing. I downgraded back to 2.4.20-gentoo-r7 after 4 hours of playing with aa-sources, and gentoo sources and various combinations of configurations with and without himem support.

This is really confusing. All kernels left memory high as reported by gkrellm2 and free. Except 2.4.20. That one used 890mb with the kernel configured for no highmem support at all so I lose 100mb or so of top ram. It has been an hour since I ran it. Free still reports 886mb used, however gkrellm2 reports 304mb used which I expect considering with highmem support my usual is 240mb.

This machine I have this trouble on has the latest of everything on it. The other machine which also uses the 2.4.20-r7 kernel, has not been updated in a while. That one also only has 512mb ram. Behavior on that machine is perfect. Memory goes up to close to 450mb, and as soon as updatedb is finished, it drops right back down to 145mb where it normally is. What appeared to be a simple case of finding which version kernel this began with, has suddenly turned into a possible shared library/gcc/whoknowswhat nightmare.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Tue Jan 20, 2004 11:13 am    Post subject: Reply with quote

I also think that when using 2.4.20 kernel my memory usage was lower, but I do not have this kernel anymore. I am ruuning gkrellm and to my recollection the used memory bar was showing about half, but now it is close to full.

BTW, in my experience, gkrellm and free alway gave the same (within 5 MB or so).
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 12:55 pm    Post subject: Reply with quote

mhodak wrote:
I also think that when using 2.4.20 kernel my memory usage was lower, but I do not have this kernel anymore. I am ruuning gkrellm and to my recollection the used memory bar was showing about half, but now it is close to full.

BTW, in my experience, gkrellm and free alway gave the same (within 5 MB or so).


I don't have the ebuild stuff any more I just tarred up my directory from the other machine. If you want it I can put it on a site you can download it. It is 200mb because I did not clear out the object modules since I didnt want to disturb the config for that machine. The free/gkrellm agreement has always been my experience too. but this time around it is different. Currently my gkrellm is telling me I have 288mb used yet my free command is way up. here it is:

total used free shared buffers cached
Mem: 902584 894704 7880 0 350996 249268
-/+ buffers/cache: 294440 608144
Swap: 498004 6968 491036

VERY odd behavior.

Chuck
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Tue Jan 20, 2004 9:39 pm    Post subject: Reply with quote

I just did tests on RH machine which uses 2.4.20 kernel and here are the results (the machine is freshly rebooted):

Code:

# free -m
             total       used       free     shared    buffers     cached
Mem:           249         43        206          0          5         20
-/+ buffers/cache:      16        233
Swap:          517          0        517

#updatedb

# free -m
             total       used       free     shared    buffers     cached
Mem:           249        229         20          0         64         22
-/+ buffers/cache:     142        107
Swap:          517          0        517


Thus it eats about half of the memory, the same as 2.4.22 kernel on my gentoo machine. But doing anything intensive (installing rpm for new kernel, running nvidia-installer) frees up most of the memory. Here is memory after "nvidia-installer -f"

Code:

#free -m
             total       used       free     shared    buffers     cached
Mem:           249        210         38          0         64        107
-/+ buffers/cache:         38        210
Swap:          517          0        517


I have also tried rerunning of upgradedb at this point to see if it wants some more memory, but memory usage stays the same after the rerun.

It seems as if the memory held by updatedb can in fact be easily freed when needed in 2.4.20, but I still do not undertsand why it is not freed immediately as it is the case for other applications (or perhaps free is wrong?). The thing is that with 2.4.22 kernel on my Gentoo box, doing emerge or something similar does not seem to free up the memory, but it may be because the gentoo box has twice the memory of RH box.

I realized that 2.4.20 kernel is still available through portage so I will try it
later today and test if its memory management is any better.

In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)?
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 9:54 pm    Post subject: Reply with quote

mhodak wrote:
<snipped for brevity>

It seems as if the memory held by updatedb can in fact be easily freed when needed in 2.4.20, but I still do not undertsand why it is not freed immediately as it is the case for other applications (or perhaps free is wrong?). The thing is that with 2.4.22 kernel on my Gentoo box, doing emerge or something similar does not seem to free up the memory, but it may be because the gentoo box has twice the memory of RH box.

I realized that 2.4.20 kernel is still available through portage so I will try it
later today and test if its memory management is any better.

In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)?


I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.

btw, I found that acpi and 2.4.20 and highmem and nvidia to not co-exist well at all. I am running the kernel without highmem, using the default 3gb split, and without acpi. It appears the acpi code in that kernel is a bit strange. I lose the use of about 100mb ram that way, but everything appars to be more than good, and even though free still is reporting 890mb used after 15 hrs, gkrellm is reporting presently 310mb used which would be very close to my expectations for what I have running at the moment.


Chuck
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Tue Jan 20, 2004 10:10 pm    Post subject: Reply with quote

RioFL wrote:
mhodak wrote:

In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)?


I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.
Chuck


Which kernel is this? Did you see (in free or gkrellm) that the amount of used memory decreased when application was done?
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Tue Jan 20, 2004 10:13 pm    Post subject: Reply with quote

mhodak wrote:
RioFL wrote:
mhodak wrote:

In the meantime, can anybody test this freeing behavior, that is to run a big job (emerge, open office, anything that requires lots of memory) and see if memory was freed (without increasing swap usage)?


I have done that and yes I can run anything I want and it appears to have enough memory without resorting to swap, I just don't like not being able to see what is real. Having memory reporting utilities is useless with this setup.
Chuck


Which kernel is this? Did you see (in free or gkrellm) that the amount of used memory decreased when application was done?


everything but updatedb reduces memory immediately upon exit. this is kernel 2.4.20-gentoo-r7

I went back to that after running 2.4.22 and having this memory problem.
Back to top
View user's profile Send private message
___c___
n00b
n00b


Joined: 04 Dec 2003
Posts: 61

PostPosted: Wed Jan 21, 2004 1:01 am    Post subject: Reply with quote

Code:


$ uname -a

Linux GEN2 2.4.22-gentoo-r5 #1 Thu Jan 15 21:30:08 CET 2004 i686 AMD Athlon(tm) processor AuthenticAMD GNU/Linux

$ grep "hda3" /etc/fstab

/dev/hda3               /               reiserfs        noatime                 0 0

$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        108        647          0          6         47
-/+ buffers/cache:         55        700
Swap:          517          0        517

$ updatedb
$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        714         41          0        326         51
-/+ buffers/cache:        335        420
Swap:          517          0        517

$ ooffice
$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        735         20          0        288        108
-/+ buffers/cache:        338        417
Swap:          517          0        517


Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Wed Jan 21, 2004 1:11 am    Post subject: Reply with quote

___c___ wrote:
Code:


$ uname -a

Linux GEN2 2.4.22-gentoo-r5 #1 Thu Jan 15 21:30:08 CET 2004 i686 AMD Athlon(tm) processor AuthenticAMD GNU/Linux

$ grep "hda3" /etc/fstab

/dev/hda3               /               reiserfs        noatime                 0 0

$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        108        647          0          6         47
-/+ buffers/cache:         55        700
Swap:          517          0        517

$ updatedb
$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        714         41          0        326         51
-/+ buffers/cache:        335        420
Swap:          517          0        517

$ ooffice
$ free -m

             total       used       free     shared    buffers     cached
Mem:           756        735         20          0        288        108
-/+ buffers/cache:        338        417
Swap:          517          0        517




presently my gkrellm is at 326mb but i have a lot going on. however free has never changed from last night, at least not any noticable amount:
gndmstr@eron gndmstr $ free -m
total used free shared buffers cached
Mem: 881 874 7 0 80 465
-/+ buffers/cache: 327 553
Swap: 486 11 475


i have 1024mb ram but with himem turned off 881 (900) is what the kernel sees. as you can see, free still sees 874mb used, but gkrellm reports 326 which is more in keeping with what i am doing.
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Wed Jan 21, 2004 1:12 am    Post subject: Reply with quote

Oh, I don't think I ever mentioned, I am running reiserfs on everything but /boot which is ext2

Chuck
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Wed Jan 21, 2004 4:34 am    Post subject: Reply with quote

Guys,
I installed 2.4.20 kernel on my Gentoo box and compared it to 2.4.22 kernel. 2.4.20 works better with respect to the updatedb problem. First running updatedb eats less memory (but it still does not release it properly) but when trying after running emerge most of the memory is released. For 2.4.22 kernel running emerge worsens the situation.

I am about to fill a bug report, do not know what else to do.
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Wed Jan 21, 2004 5:54 am    Post subject: Reply with quote

Bug submitted as # 38886
https://bugs.gentoo.org/show_bug.cgi?id=38886

Please add your comments so that developers see that I am not the only one that sees the problem.
Back to top
View user's profile Send private message
ekoontz
n00b
n00b


Joined: 18 Apr 2002
Posts: 67
Location: San Francisco, California

PostPosted: Sat Jan 24, 2004 9:06 am    Post subject: Reply with quote

I had this problem also; not knowing what else to do, I tried 2.6.1-rc1, and it seems to have gone away.

I know that doesn't help much with finding out the root cause, but I guess it qualifies as a workaround! :)
_________________
In Soviet Gentoo, portage emerges -u!
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Sat Jan 24, 2004 11:55 am    Post subject: Reply with quote

Great to hear that there is a kernel in which this weird memory behavior was fixed !!!
Back to top
View user's profile Send private message
tomaw
Guru
Guru


Joined: 26 Mar 2003
Posts: 429
Location: UK

PostPosted: Sat Jan 24, 2004 11:57 am    Post subject: Reply with quote

I seem to still suffer from the memory reporting being wrong in 2.6.1 final - hoping to try 2.6.2 within a few days and see if that has gone anywhere to fixing it...
_________________
Tom Wesley
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Sat Jan 24, 2004 12:01 pm    Post subject: Reply with quote

ekoontz wrote:
I had this problem also; not knowing what else to do, I tried 2.6.1-rc1, and it seems to have gone away.

I know that doesn't help much with finding out the root cause, but I guess it qualifies as a workaround! :)


never played with the dev sources before because I always thought that they were supposed to be too unstable... i see bugs images on 2.6.1-r1, don't find an rc1, but there is a 2.6.1 that looks like a 'release' so I am going to try that one. 2.4.20 is just too 'backward' for me. I notice it is slightly slower than 2.4.22, video isn't as crisp and of course memory management is worse with buggy highmem code.. so I am going to try the 2.6.1 this morning and see what it does.

My biggest concern is if I switch to this, can I still go backward to lower versions? I heard somewhere that 2.6.1 did away with devfs which is the core of my system.

Chuck
Back to top
View user's profile Send private message
ekoontz
n00b
n00b


Joined: 18 Apr 2002
Posts: 67
Location: San Francisco, California

PostPosted: Sat Jan 24, 2004 5:16 pm    Post subject: Reply with quote

RioFL, devfs seems to be considered deprecated in 2.6, but still you can choose it (as I did), and it seems to work fine.

You can still boot older kernels; devfs doesn't change anything permanently about the /dev directory as far as I Know. (it just creates /dev entries dynamically which go away at the next boot).

tomaw, here's some data; i've been up for 4 days so updatedb has been running every night through cron..
(the load is high because I'm emerging a bunch of stuff :) )
Code:


hiro-tan(ekoontz)[~] $ uptime
 09:15:25 up 4 days,  8:11,  2 users,  load average: 1.01, 1.07, 1.06
hiro-tan(ekoontz)[~] $ free -m
             total       used       free     shared    buffers     cached
Mem:           216        203         12          0         21         85
-/+ buffers/cache:         96        120
Swap:         1027         60        967
hiro-tan(ekoontz)[~] $ uname -a
Linux hiro-tan.org 2.6.1-rc1 #1 SMP Fri Jan 16 00:04:54 PST 2004 i686 VIA Samuel 2 CentaurHauls GNU/Linux

_________________
In Soviet Gentoo, portage emerges -u!
Back to top
View user's profile Send private message
RioFL
Guru
Guru


Joined: 31 Oct 2002
Posts: 407

PostPosted: Sat Jan 24, 2004 6:25 pm    Post subject: Reply with quote

ekoontz wrote:
RioFL, devfs seems to be considered deprecated in 2.6, but still you can choose it (as I did), and it seems to work fine.

You can still boot older kernels; devfs doesn't change anything permanently about the /dev directory as far as I Know. (it just creates /dev entries dynamically which go away at the next boot).

tomaw, here's some data; i've been up for 4 days so updatedb has been running every night through cron..
(the load is high because I'm emerging a bunch of stuff :) )
Code:


hiro-tan(ekoontz)[~] $ uptime
 09:15:25 up 4 days,  8:11,  2 users,  load average: 1.01, 1.07, 1.06
hiro-tan(ekoontz)[~] $ free -m
             total       used       free     shared    buffers     cached
Mem:           216        203         12          0         21         85
-/+ buffers/cache:         96        120
Swap:         1027         60        967
hiro-tan(ekoontz)[~] $ uname -a
Linux hiro-tan.org 2.6.1-rc1 #1 SMP Fri Jan 16 00:04:54 PST 2004 i686 VIA Samuel 2 CentaurHauls GNU/Linux



I put it in and am not using devfs. Decided to get into the spirit of 2.6.1 and 'go with the flow'. Everything appears to work just fine except my sensors and my logitech quickcam module which won't compile now. They eliminated LM80 sensors although the VIAPro stuff is in there, gkrellm isnt seeing the sensors at all:( will have to research it. Outside of that, my video is better, everything seems quicker.

Unfortunately the updatedb problem is still there so for now I have decided to eliminiate it from the nightly run and I will run it manually then reboot right after.. maybe a few times a week.

Only other problem I see which isnt really a problem, is the boot script warns me devfs is not installed, but it does not appear to affect anything.

Chuck
Back to top
View user's profile Send private message
mhodak
Veteran
Veteran


Joined: 15 Nov 2003
Posts: 1218

PostPosted: Thu Jan 29, 2004 11:48 am    Post subject: Reply with quote

Guys,
I checked Kernel mailing list to see if there were any replies to the post with our problem, but there are none, which means that there is no solutiom or explanation for the problem we are experiencing. Considering that this exists even in 2.6 kernels I would really like to see an explanation from somebody who knows about kernel memory management.
I am thinking about posting there our problem again, it may get noticed.

On the other hand, my machine has 10 days uptime now, I do updatedb everyday and besides having almost 100% used memory (reported by gkrellm), it does not seem to affect me.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2, 3, 4
Page 4 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum