Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge's appetite for memory..
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Draxon
n00b
n00b


Joined: 24 Feb 2003
Posts: 21

PostPosted: Sat Apr 05, 2003 1:50 pm    Post subject: Emerge's appetite for memory.. Reply with quote

Hi.. I have 512mb physical memory in my machine, and a 500 mb swap space. The other day I was updating world, which worked fine.. but afterward I only had 3mb of memory listed as 'free'! I thought maybe it was just some data getting cached by gcc, but it didn't go away, and my system took a huge performance hit. Not only that I began getting 'low/no memory' errors from several applications. I dug around the forums here a bit looking for other people with similar problems but I didn't find any that matched mine exactly. I saw mention of a bug with the nvidia drivers, but I don't have an nvidia card, nor do I have the glx stuff installed.

I thought maybe one of my daemons had a memory leak, so I shut them all off and rebooted. Once booted up memory was fine (had about 500mb free) but then I re-emerged something (less I think.. if it matters) and my memory went from 500 free to 388 free. I figured the kernel might just be holding onto stuff until more memory is needed, so I proceeded to emerge a bunch more stuff to suck up memory, then I attempted to start a couple daemons and they failed to start due to lack of memory. Here is the output of free after starting up with no non-essential daemon processes, and then emerge -u'ing less:
Code:
lnr1 etc # free -m
             total       used       free     shared    buffers     cached
Mem:           502        113        388          0          0         85
-/+ buffers/cache:         28        474
Swap:          442          0        442


My system is a dual Piii 866 machine with 512mb RAM, and something like 340gigs of storage over 3 disks. I use xfs, and am running the 2.4.20-xfs-r2 kernel.

Any ideas?

Thanks in advance..


-Matt
Back to top
View user's profile Send private message
bsolar
Bodhisattva
Bodhisattva


Joined: 12 Jan 2003
Posts: 2764

PostPosted: Sat Apr 05, 2003 1:59 pm    Post subject: Reply with quote

Can you post also the output of 'free' when experiencing your problem?
_________________
I may not agree with what you say, but I'll defend to the death your right to say it.
Back to top
View user's profile Send private message
Draxon
n00b
n00b


Joined: 24 Feb 2003
Posts: 21

PostPosted: Sat Apr 05, 2003 2:09 pm    Post subject: Reply with quote

This happened after I uploaded a large file ~800mb via my FTP daemon.. maybe proftpd has a memory leak? ... although I wasn't using proftpd when I first discovered the problem
Code:
             total       used       free     shared    buffers     cached
Mem:           502        494          7          0          0        448
-/+ buffers/cache:         45        456
Swap:          442          0        442


Let me reboot the machine and attempt to duplicate the conditions I mentioned earlier. (I'll try use up all the memory with emerge) I'll post free's output in a moment..
Back to top
View user's profile Send private message
Draxon
n00b
n00b


Joined: 24 Feb 2003
Posts: 21

PostPosted: Sat Apr 05, 2003 2:28 pm    Post subject: Reply with quote

Code:
emerge rrdtool; emerge proftpd;

Before:
Code:
             total       used       free     shared    buffers     cached
Mem:           502         52        449          0          0         28
-/+ buffers/cache:         24        477
Swap:          442          0        442

After:
Code:
             total       used       free     shared    buffers     cached
Mem:           502        116        386          0          0         84
-/+ buffers/cache:         31        470
Swap:          442          0        442

.. this doesn't exactly illustrate what I was talking about but I don't know of a good way to go back and find all the packages that were merged when I updated world a few days ago...

Perhaps proftpd was being used while I was emerging? But why would proftpd suck up all the memory..
Back to top
View user's profile Send private message
elzbal
Guru
Guru


Joined: 31 Aug 2002
Posts: 364
Location: Seattle, WA, USA

PostPosted: Sat Apr 05, 2003 3:51 pm    Post subject: Reply with quote

Are you looking at the first line, or at the '+/- buffers...' line? Keep in mind that, to most modern operating systems, unused memory is wasted memory and memory is much better employed in keeping a healthy disk cache and buffering anything that can be buffered. To see the amount of physical memory that is available to processes, take a look at the '+/- buffers...' line.

Quote:

Before:
Code:

             total       used       free     shared    buffers     cached
Mem:           502         52        449          0          0         28
-/+ buffers/cache:         24        477
Swap:          442          0        442

After:
Code:

Code:
             total       used       free     shared    buffers     cached
Mem:           502        116        386          0          0         84
-/+ buffers/cache:         31        470
Swap:          442          0        442


Here, it shows you had 477MB free before, 470MB free after. 'free' does show a small increase in used memory, 7MB in this case. Can you account for that by looking at 'ps -aux' and looking at the 'rss' column? That should show what is using more memory.

Here is what I came up with, by the way, when testing an emerge:

Code:

# free -m > free.txt && emerge proftpd && free -m >> free.txt
.....
# cat free.txt
             total       used       free     shared    buffers     cached
Mem:           756        345        410          0         18        136
-/+ buffers/cache:        191        564
Swap:          509          4        505
             total       used       free     shared    buffers     cached
Mem:           756        348        407          0         19        137
-/+ buffers/cache:        191        564
Swap:          509          4        505


In my case, there is some additional cache used (410 vs 407MB, but there is no decrease in the memory available to processes (564MB).

Please continue testing. If you are still getting the same results, let us know and you may want to consider reporting a bug.
Back to top
View user's profile Send private message
Draxon
n00b
n00b


Joined: 24 Feb 2003
Posts: 21

PostPosted: Sun Apr 06, 2003 2:52 am    Post subject: Reply with quote

Hmm.. yea I never really worried about how much memory was reported free in the past because I thought it worked as you just described (not saying it doesn't). The only reason I thought it might matter is after I began getting not enough memory errors by a couple applications, but now I'm thinking there must have been something else in the mix before. I'm not recieving memory errors any longer.. I guess I should have dug a bit deeper when I was getting errors.

Thanks for your help.
Back to top
View user's profile Send private message
gillesg
Tux's lil' helper
Tux's lil' helper


Joined: 17 Sep 2002
Posts: 90

PostPosted: Mon Apr 21, 2003 7:32 pm    Post subject: I have the same Reply with quote

Hi,

Reinstalling a Gentoo 1.4 on a new Hard drive, I noticed exactly the same behavior.
Just booted the system says thru top :
used : 20000k free : 730000k

I run : # emerge kde
It fetches and compiles xfree and kde.
After 1 hour (xfree not yet compiled), top shows :
used : 706000k free : 50000k

That is so weird. Where all the available memory is gone, knowing that there is almost no process running (system installation).

Any hint to go further ?

Gilles
Back to top
View user's profile Send private message
elzbal
Guru
Guru


Joined: 31 Aug 2002
Posts: 364
Location: Seattle, WA, USA

PostPosted: Mon Apr 21, 2003 8:01 pm    Post subject: Re: I have the same Reply with quote

gillesg wrote:
Hi,

Reinstalling a Gentoo 1.4 on a new Hard drive, I noticed exactly the same behavior.
Just booted the system says thru top :
used : 20000k free : 730000k

I run : # emerge kde
It fetches and compiles xfree and kde.
After 1 hour (xfree not yet compiled), top shows :
used : 706000k free : 50000k

That is so weird. Where all the available memory is gone, knowing that there is almost no process running (system installation).

Any hint to go further ?

Gilles


What does 'free -m' show? I would imagine a lot of that is taken up in cache, especially during heavy disk use such as you get during compilation. The memory being used for cache and buffers is reported as 'used' by top, but is readily available to processes upon request.
Back to top
View user's profile Send private message
hucksteroony
n00b
n00b


Joined: 04 Jan 2003
Posts: 2

PostPosted: Wed Apr 23, 2003 8:08 am    Post subject: Reply with quote

I'm getting a similar problem whilst trying to get scrollkeeper emerged (dtd/sgml/xml emerging problems that belong somewhere else).

I'm emerging a set of docbook, xml,sgml packages and all starts well. After a while the hard drive starts going berserk and the system slows down (no X, just console sessions in use).

From memory (mine that is) free would show only about 50-60Mb used to begin with. When the disk starts "thrashing" this is what I see with free -m and top....

# free -m
total used free shared buffers cached
Mem: 622 617 4 0 0 2
-/+ buffers/cache: 614 7
Swap: 980 29 951


top - 21:03:41 up 3:23, 2 users, load average: 1.34, 1.20, 1.18
Tasks: 32 total, 1 running, 31 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% user, 1.9% system, 0.0% nice, 97.9% idle
Mem: 637636k total, 632860k used, 4776k free, 132k buffers
Swap: 1004052k total, 30452k used, 973600k free, 3244k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command
19549 root 15 0 25260 4616 4368 D 1.0 0.7 0:57.30 cc1plus
5 root 15 0 0 0 0 S 0.3 0.0 0:32.65 kswapd
19642 root 16 0 432 368 328 R 0.3 0.1 0:00.01 top
1 root 15 0 80 24 20 S 0.0 0.0 0:04.70 init
2 root 15 0 0 0 0 S 0.0 0.0 0:00.01 keventd
3 root 15 0 0 0 0 S 0.0 0.0 0:00.01 kapmd
4 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU0
6 root 15 0 0 0 0 S 0.0 0.0 0:00.00 bdflush
7 root 15 0 0 0 0 S 0.0 0.0 0:00.02 kupdated
9 root 25 0 0 0 0 S 0.0 0.0 0:00.00 khubd
12 root 15 0 0 0 0 S 0.0 0.0 0:00.60 kjournald
36 root 15 0 300 0 0 S 0.0 0.0 0:00.01 devfsd

As you can see, not a lot of processing going on. Previously, when I've killed the emerge, the memory doesn't "get released".

I've been playing with Linux for a while now and this is a first for me. (Duron 750, 640Mb RAM, 1Gb SWAP).

Angus.
Back to top
View user's profile Send private message
hucksteroony
n00b
n00b


Joined: 04 Jan 2003
Posts: 2

PostPosted: Thu Apr 24, 2003 12:08 am    Post subject: Reply with quote

I did an upgrade from ck-sources 2.4.20 r5 to r6 and I haven't experienced the same disk thrashing as before (and I have been doing quite a bit of emerging).



Angus.

:)
_________________
Infamy! Infamy! They've all got it in for me.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum