memory leak in portage?

suineg · Posted: Mon Feb 28, 2005 2:53 am Post subject: memory leak in portage?

I think I may have found a memory leak in portage, I noticed it for sure with an emerge sync, not sure about the rest of portage.

basically, I am running nothing but xfce4, it consumes about 14% of my memory (512) with gdm running etc. then today after an emerge sync, it jumped up to 19% and stayed there, a few hours later I decided to test my theory by running another emerge sync, this time it jumped up to 33% and still refuses to go down.

whats going on here!? is there a leak in portage? or do I have some other problem with my system.

I am just using torsmo for my readings, which is pretty accuarate afaik.

I can't stand wasted memory

any ideas?

_never_ · Posted: Mon Feb 28, 2005 3:10 am Post subject:

Don't worry, this isn't wasted memory. =)

Linux tries to "remember" as much as possible to speed up hardware access. It has disk caches and other stuff, which all count to used memory. It is freed as soon as other, more recent data arrives. If you want Linux to try to have a constant amount of free memory at all times, then you might want to echo some numeric value to /proc/sys/vm/min_free_kbytes:

SerfurJ · l33t Joined: 10 Apr 2004 Posts: 824 Location: Texas

if there's a memory leak, it would be freed up when the process is done. you're not looking at cached memory are you? look at the second row of output from "free".

suineg · Posted: Mon Feb 28, 2005 3:41 am Post subject:

It's not disk cache, because torsmo automatically subtracts your disk cache to get the actual used memory. All processes relating to portage that I can tell, are finished yet I still have this excess used memory.

suineg · Posted: Mon Feb 28, 2005 3:43 am Post subject:

I checked free just to be sure and sure enough, if I had been looking at cached memory, my memory usage would be something more like 60%

_never_ · Posted: Mon Feb 28, 2005 3:47 am Post subject:

suineg · Posted: Mon Feb 28, 2005 3:52 am Post subject:

_never_ · Posted: Mon Feb 28, 2005 4:04 am Post subject:

If you are sure, then look at top (as root, if you use grsecurity) and identify the process that is leaking so much memory. Maybe it's your terminal emulator or the X server - or torsmo itself. =D

Well, serious: If it's really not the cache, try to identify the process that is responsible for it. If it's your terminal emulator, then everything is fine - in that case it's the scrollback. Sort by RAM usage in top and watch it while a emerge sync is running.
_________________
Knowledge is Power.

suineg · Posted: Mon Feb 28, 2005 4:21 am Post subject:

okay, I did as you suggested, and my terminal mem usage did not increase during the sync at all. emerge jumps way up to about 8% this time however all but 1% of the increase was freed.

at this point you might think i'm crazy, but i don't know where else this memory could be going to , I definately noticed the fact that it didn't drop back down the last two times I did an emerge sync (now three), I will try it again as soon as portage lets me

SerfurJ · l33t Joined: 10 Apr 2004 Posts: 824 Location: Texas

if you start "top", then press "M", you can see very easily what's using memory under the "RES" column (resident memory).

_never_ · Posted: Mon Feb 28, 2005 4:48 am Post subject:

Maybe it was the cache. And now everything is already in the cache, so it doesn't need to load it again. =)

There is something you could do to test it: try to fill the memory with something. For example with this command:

suineg · Posted: Mon Feb 28, 2005 5:04 am Post subject:

right, so according to top, I am using ~136mb, while torsmo/free claim that I am using 189mb, there seems to be a missing 11+ percent there.

if I close firefox, which I didn't have running when I read the 19% usage, I am using about 19% plus I have gaim conversations open which I didn't before so that could easily account for the extra ~4mb I am missing (I think). so this would seem about right.

everything that I am seeing points me to believe that I have lost at least 11% of my ram to nothingness.

suineg · Posted: Mon Feb 28, 2005 5:10 am Post subject:

_never_ · Posted: Mon Feb 28, 2005 5:18 am Post subject:

You're welcome.

One source for this problem could be the kernel processes like pdflush and the various processes for your filesystem types. But they don't seem to consume any userspace memory - their memory consuption seems to be shown as cache. I don't know exactly.

Edit: as cache or as buffers.
_________________
Knowledge is Power.

suineg · Posted: Mon Feb 28, 2005 5:19 am Post subject:

so maybe a problem with reiser4 not properly showing disk cache as cache?

_never_ · Posted: Mon Feb 28, 2005 5:22 am Post subject:

Well, I guess, it's not a problem at all. =)

I don't know how stable ReiserFS 4 is. Still on ReiserFS 3 here - I'm yet too paranoid for the next version. =)
_________________
Knowledge is Power.

suineg · Posted: Mon Feb 28, 2005 5:29 am Post subject:

I can't easily check an older version of torsmo as version .17 won't compile on my machine anymore for some reason.

it seems unlikely that it would be torsmo anyway

if it were a reiser4 problem, would it be related to the kernel driver or something from the reiser4progs package?
I haven't updated reiser4progs in forever (if ever since I last installed).

but I have switched kernels recently.

one thing I do know for certain is that I didn't used to have this problem, so there MUST be a way to fix it

suineg · Posted: Mon Feb 28, 2005 5:31 am Post subject:

_never_ · Posted: Mon Feb 28, 2005 5:34 am Post subject:

suineg · Posted: Mon Feb 28, 2005 8:37 am Post subject:

okay, still trying to remedy this, I ran another test, I rebooted, check my ram with free -m. and under the +- buffers/cache I get 93mb. then I run an emerge sync, and check it after it finishes, and my new value, under the same section is 194.

I believe I have the latest reiser4 patches, so I'm really stumped.

not trying to wear you out, but I really would like this "problem" fixed

_never_ · Posted: Mon Feb 28, 2005 8:46 am Post subject:

This is totally normal. The buffers get filled and filled, until they reach the memory limits. But as I said, they never get swapped out. Don't worry, this is not taking any memory. If memory is needed, then the oldest records in the buffers/cache are deleted.
_________________
Knowledge is Power.

suineg · Posted: Mon Feb 28, 2005 8:56 am Post subject:

_never_ · Posted: Mon Feb 28, 2005 9:05 am Post subject:

suineg · Posted: Mon Feb 28, 2005 9:07 am Post subject:

okay I guess I will have to be content with that.

thanks again.

_never_ · Posted: Mon Feb 28, 2005 9:13 am Post subject: