Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
memory leak in portage?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 2:53 am    Post subject: memory leak in portage? Reply with quote

I think I may have found a memory leak in portage, I noticed it for sure with an emerge sync, not sure about the rest of portage.

basically, I am running nothing but xfce4, it consumes about 14% of my memory (512) with gdm running etc. then today after an emerge sync, it jumped up to 19% and stayed there, a few hours later I decided to test my theory by running another emerge sync, this time it jumped up to 33% and still refuses to go down.

whats going on here!? is there a leak in portage? or do I have some other problem with my system.

I am just using torsmo for my readings, which is pretty accuarate afaik.

I can't stand wasted memory :)
any ideas?
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 3:10 am    Post subject: Reply with quote

Don't worry, this isn't wasted memory. =)

Linux tries to "remember" as much as possible to speed up hardware access. It has disk caches and other stuff, which all count to used memory. It is freed as soon as other, more recent data arrives. If you want Linux to try to have a constant amount of free memory at all times, then you might want to echo some numeric value to /proc/sys/vm/min_free_kbytes:

Code:
echo 1024 > /proc/sys/vm/min_free_kbytes


But be careful. You can easily drop Linux into an endless loop, if you set this value too high. If you want to read the current value, use this command:

Code:
cat /proc/sys/vm/min_free_kbytes


You might also play around with /proc/sys/vm/swappiness. This value controls (in percent), how much Linux likes to swap data. The higher, the more gets into swap and the later it gets out again. The lower, the less Linux likes to swap and it gets back to real memory sooner. Be warned: never set this to 0, else your system will crash sooner or later - as soon as Linux has to swap something.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
SerfurJ
l33t
l33t


Joined: 10 Apr 2004
Posts: 824
Location: Texas

PostPosted: Mon Feb 28, 2005 3:13 am    Post subject: Reply with quote

if there's a memory leak, it would be freed up when the process is done. you're not looking at cached memory are you? look at the second row of output from "free".
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 3:41 am    Post subject: Reply with quote

It's not disk cache, because torsmo automatically subtracts your disk cache to get the actual used memory. All processes relating to portage that I can tell, are finished yet I still have this excess used memory.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 3:43 am    Post subject: Reply with quote

I checked free just to be sure and sure enough, if I had been looking at cached memory, my memory usage would be something more like 60%
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 3:47 am    Post subject: Reply with quote

suineg wrote:
It's not disk cache, because torsmo automatically subtracts your disk cache to get the actual used memory. All processes relating to portage that I can tell, are finished yet I still have this excess used memory.


If this was true, my whole memory would be leaked. =)
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 3:52 am    Post subject: Reply with quote

_never_ wrote:
suineg wrote:
It's not disk cache, because torsmo automatically subtracts your disk cache to get the actual used memory. All processes relating to portage that I can tell, are finished yet I still have this excess used memory.


If this was true, my whole memory would be leaked. =)


This actualy IS true, and can be enabled/disabled from your .torsmorc with this option:

Code:
# Subtract file system buffers from used memory?
no_buffers yes


so I am most positive I am seeing the correct numbers, plus using free to verify the torsmo numbers, subtracting disk cache myself, it would seem that torsmo is accurate.
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 4:04 am    Post subject: Reply with quote

If you are sure, then look at top (as root, if you use grsecurity) and identify the process that is leaking so much memory. Maybe it's your terminal emulator or the X server - or torsmo itself. =D

Well, serious: If it's really not the cache, try to identify the process that is responsible for it. If it's your terminal emulator, then everything is fine - in that case it's the scrollback. Sort by RAM usage in top and watch it while a emerge sync is running.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 4:21 am    Post subject: Reply with quote

okay, I did as you suggested, and my terminal mem usage did not increase during the sync at all. emerge jumps way up to about 8% this time however all but 1% of the increase was freed.

at this point you might think i'm crazy, but i don't know where else this memory could be going to , I definately noticed the fact that it didn't drop back down the last two times I did an emerge sync (now three), I will try it again as soon as portage lets me :)
Back to top
View user's profile Send private message
SerfurJ
l33t
l33t


Joined: 10 Apr 2004
Posts: 824
Location: Texas

PostPosted: Mon Feb 28, 2005 4:40 am    Post subject: Reply with quote

if you start "top", then press "M", you can see very easily what's using memory under the "RES" column (resident memory).
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 4:48 am    Post subject: Reply with quote

Maybe it was the cache. And now everything is already in the cache, so it doesn't need to load it again. =)

There is something you could do to test it: try to fill the memory with something. For example with this command:

Code:
(dd if=/dev/zero bs=1M count=400; read) | sort


This forcibly loads 400 MB into memory and keeps it there until the return key is pressed. If it is the cache, then at some point, when you increase the count= value, the memory usage will not increase. This means, that the kernel began forgetting the cache contents, as they are never swapped to disk. Once you press the return key, your memory then drops back to an even lower value than before, because the cache entries disappeared - if it's the cache.

Another way to test it: grab some large file, which wouldn't fit into your memory, let's say movie.avi and do the following:

Code:
cat movie.avi > /dev/null


This will do nothing more than loading the file into the cache. Look at torsmo while this is done. Does the memory usage increase? It shouldn't, if the cache is not taken into account. If you don't have such a large file, create it:

Code:
dd if=/dev/urandom of=movie.avi bs=1M count=600


Hopefully this answers your questions.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 5:04 am    Post subject: Reply with quote

right, so according to top, I am using ~136mb, while torsmo/free claim that I am using 189mb, there seems to be a missing 11+ percent there.

if I close firefox, which I didn't have running when I read the 19% usage, I am using about 19% plus I have gaim conversations open which I didn't before so that could easily account for the extra ~4mb I am missing (I think). so this would seem about right.

everything that I am seeing points me to believe that I have lost at least 11% of my ram to nothingness.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 5:10 am    Post subject: Reply with quote

_never_ wrote:
Maybe it was the cache. And now everything is already in the cache, so it doesn't need to load it again. =)

There is something you could do to test it: try to fill the memory with something. For example with this command:

Code:
(dd if=/dev/zero bs=1M count=400; read) | sort


This forcibly loads 400 MB into memory and keeps it there until the return key is pressed. If it is the cache, then at some point, when you increase the count= value, the memory usage will not increase. This means, that the kernel began forgetting the cache contents, as they are never swapped to disk. Once you press the return key, your memory then drops back to an even lower value than before, because the cache entries disappeared - if it's the cache.

Another way to test it: grab some large file, which wouldn't fit into your memory, let's say movie.avi and do the following:

Code:
cat movie.avi > /dev/null


This will do nothing more than loading the file into the cache. Look at torsmo while this is done. Does the memory usage increase? It shouldn't, if the cache is not taken into account. If you don't have such a large file, create it:

Code:
dd if=/dev/urandom of=movie.avi bs=1M count=600


Hopefully this answers your questions.


okay I tried your first suggestion, my memory usage jumps up to 91%, then after I quit, jumps back down to 18% which seems about right for my running system. This would indicate that it IS the disk cache as you had guess.

The strange this is that I just very recently noticed this, and I know that torsmo, when configured to do so is supposed to subtract disk cache.

so I am going to assume that the problem here is perhaps the latest version of torsmo.

I will look around and see if anyone else has been having such a problem with torsmo, and post back if I don't find anything.

thanks for all the help :)
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 5:18 am    Post subject: Reply with quote

You're welcome.

One source for this problem could be the kernel processes like pdflush and the various processes for your filesystem types. But they don't seem to consume any userspace memory - their memory consuption seems to be shown as cache. I don't know exactly.

Edit: as cache or as buffers.
_________________
Knowledge is Power.


Last edited by _never_ on Mon Feb 28, 2005 5:20 am; edited 1 time in total
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 5:19 am    Post subject: Reply with quote

so maybe a problem with reiser4 not properly showing disk cache as cache?
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 5:22 am    Post subject: Reply with quote

Well, I guess, it's not a problem at all. =)

I don't know how stable ReiserFS 4 is. Still on ReiserFS 3 here - I'm yet too paranoid for the next version. =)
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 5:29 am    Post subject: Reply with quote

I can't easily check an older version of torsmo as version .17 won't compile on my machine anymore for some reason.

it seems unlikely that it would be torsmo anyway

if it were a reiser4 problem, would it be related to the kernel driver or something from the reiser4progs package?
I haven't updated reiser4progs in forever (if ever since I last installed).

but I have switched kernels recently.

one thing I do know for certain is that I didn't used to have this problem, so there MUST be a way to fix it :)
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 5:31 am    Post subject: Reply with quote

_never_ wrote:
Well, I guess, it's not a problem at all. =)

I don't know how stable ReiserFS 4 is. Still on ReiserFS 3 here - I'm yet too paranoid for the next version. =)


heh, i have been using reiser4 since before it was officially released, and have had no problems whatsoever (aside from this sortof non-problem :)).

but I know some people have, so I guess that makes me lucky.

and no, I guess it wouldn't REALLY be a problem, but more like an irritation :)
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 5:34 am    Post subject: Reply with quote

suineg wrote:
if it were a reiser4 problem, would it be related to the kernel driver or something from the reiser4progs package?
I haven't updated reiser4progs in forever (if ever since I last installed).


It may be the progs, if you ever used them. But mostly it's the driver.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 8:37 am    Post subject: Reply with quote

okay, still trying to remedy this, I ran another test, I rebooted, check my ram with free -m. and under the +- buffers/cache I get 93mb. then I run an emerge sync, and check it after it finishes, and my new value, under the same section is 194.

I believe I have the latest reiser4 patches, so I'm really stumped.

not trying to wear you out, but I really would like this "problem" fixed :)
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 8:46 am    Post subject: Reply with quote

This is totally normal. The buffers get filled and filled, until they reach the memory limits. But as I said, they never get swapped out. Don't worry, this is not taking any memory. If memory is needed, then the oldest records in the buffers/cache are deleted.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 8:56 am    Post subject: Reply with quote

_never_ wrote:
This is totally normal. The buffers get filled and filled, until they reach the memory limits. But as I said, they never get swapped out. Don't worry, this is not taking any memory. If memory is needed, then the oldest records in the buffers/cache are deleted.


I know that buffer swapping etc is normal and all that, just to make sure I wasn't mistaken I refreshed my memory using this

http://forums.gentoo.org/viewtopic.php?t=175419&highlight=memory

which is a pretty good post.

according to that, you can read the number minus the biffers and cache, which should be the value actually being used by running processes etc, using free -m and checking out the +/- buffers/cache part.

that is the number i'm reading, and it still appears as if there were nothing wrong.

If it were just a buffer/cache problem, wouldn't other programs have this "problem"?

i've tested almost every app that I have and none of them have this same problem.

I know this may not be important, but its nice to see how much ram is actually free, so that for example when I go to run a game, I need to know whether or not I should close some of the apps I have open or whatever.
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 9:05 am    Post subject: Reply with quote

suineg wrote:
according to that, you can read the number minus the biffers and cache, which should be the value actually being used by running processes etc, using free -m and checking out the +/- buffers/cache part.


Exactly. But remember that kernel processes are also shown.

suineg wrote:
that is the number i'm reading, and it still appears as if there were nothing wrong.


There is nothing wrong. =)

suineg wrote:
If it were just a buffer/cache problem, wouldn't other programs have this "problem"?


That's the point. It isn't a problem.

suineg wrote:
I know this may not be important, but its nice to see how much ram is actually free, so that for example when I go to run a game, I need to know whether or not I should close some of the apps I have open or whatever.


Don't worry about this. Active programs always get the highest memory priority. So if there should ever swapping be needed, the game gets more memory than any other process, since the other processes are idle (if they are). And with idle I don't mean the CPU power used. For example the X server uses lots of RAM, but most of it is swapped out all the time, because the server doesn't actually use it.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
suineg
Apprentice
Apprentice


Joined: 02 Mar 2004
Posts: 200
Location: Los Angeles

PostPosted: Mon Feb 28, 2005 9:07 am    Post subject: Reply with quote

okay I guess I will have to be content with that.

thanks again.
Back to top
View user's profile Send private message
_never_
Apprentice
Apprentice


Joined: 10 Jun 2004
Posts: 285
Location: BW, Germany

PostPosted: Mon Feb 28, 2005 9:13 am    Post subject: Reply with quote

suineg wrote:
okay I guess I will have to be content with that.

thanks again.


Finally! =D

No problem.
_________________
Knowledge is Power.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum