Linux Memory Management or 'Why is there no free RAM?'

[log]

After I upgraded to 1GB and enabled high memory, I did not notice a drop in performance. Although the costomised software im using uses more ram than CPU...oh well, im happy

[sapphirecat]

[i_hate_your_os]

The .88G limitation was news to me. Great thread, thanks!
_________________
-IHYOS

"All laws which are repugnant to the constitution are null and void."
-Marbury vs. Madison, 5 US (2 Cranch) 137, 174, 176, (1803)

[nadamsieee]

[tommy_fila]

For everyone who is wondering why their swappiness settings are always changing to seemingly random numbers:

Direct quote from bk0

[Chonhulio]

[Clansman]

hi everyone.

excelent post by sapphirecat!

yet, i think few people that posted here got the original post's point according to the post's replies... overall, about the memory size and occupancy one can conclude:

Having low free memory reported from 'top' DOES NOT MEAN that your system is desperately and unrecoverably occupying all of your memory (or most of it), meaning that you must rush to the nearest and most expensive store to buy another Gb of ram.

The linux kernel has a memory management policy that cleverly tries to use up as much memory as it possibly can, to improve performance. This performance improvement can be very visible if that used memory becomes disk cache because as stated before, hard drive acesses are about 1000 times slower than RAM, so getting disk's data (cache) from ram is very fast. The RAM is the fastest and second biggest memory available outside the cpu.

That big memory cache forcedly used by the linux kernel to avoid wasted memory can be VERY QUICKLY shrinked up to allow user applications to use memory/more memory.

So, having 70Mb of free ram when you have only 1Gb is not a bad thing and does not mean you have to buy new memory to boost performance. You would probably find out that with your new 2Gb of ram, you'd end up having... 100Mb free...

[]

[sanity]

I'm still not getting what auto-swappiness optimizes for, but I don't use it anyway.

On all my machines, I sort of take the opposite approach. I set swappiness to 100. This is especially useful on my desktop. Consider: My brother runs GNOME. I run Fluxbox. We run them on different X servers, so that we can swtich off, leaving programs open and running, without sharing settings / giving him access to my settings (PGP key, etc. etc.)

Obviously, when he's on, he uses a lot of RAM. Now suppose I want to play ut2004. You see the problem?

Of course, his stuff can be swapped out. It's not likely to happen, though, unless I'm actually running out of memory, or unless swappiness is quite high. And if he's hogging the RAM, I don't get to use it for buffer/cache, which can really help speed sometimes. (Oops! Firefox crashed! (buggy flash plugin) Oh, but it's cached -- 1-2 seconds and I have a browser again.)

This is also helpful against fragmentation. This may be getting just a bit obsessive, but I use reiser4, and if I'm downloading a 250 meg file and I have 500 megs of RAM, it may very well stay buffered until the download is done, and then be written entirely contiguously.

Basic formula goes like this: if you think have enough RAM, you might set swappiness=0. Like, if you've got 3 gigs of RAM on a desktop machine, and you leave it on all night, cron jobs won't take significantly longer, and they won't push your apps out into swap -- thus, you come back and everything's just as you left it, no waiting around.

If you do not have enough RAM, or if you think you might be wasting some RAM, you might set swappiness=100. Now, all the things you aren't using go to swap, out of the way for things you are using, such as ut2004. The downside? After leaving the computer for awhile, it's quite likely that a cron job or a running download or something has kicked your email, your browser, even your terminal emulator out to swap, and thus, it might take 5 seconds for your computer to come back to life, and another 1-2 seconds each time you bring up a window that you haven't seen in awhile.

My last OS was Windows 98. 5 seconds I can live with, to avoid buying enough RAM to run GNOME, two instances of Fluxbox, 3 X servers, and ut2004.

[sploo22]

Along these same lines, is it possible to force a program and its assorted libraries to stay in the cache, so e.g. I can have Firefox ready to start up even if I haven't been using it recently? Preferably something more robust than "dd if=/usr/bin/firefox of=/dev/null" every 30 seconds. Does the kernel provide this level of control?
_________________
This signature will self-destruct in 10 seconds. Close browser window now to avoid permanent monitor damage.

[Clansman]

[dmitrio]

I have copied this, with permission of sapphirecat and bk0, to gentoo-wiki.com
http://gentoo-wiki.com/Linux_Memory_Management
If you see anything that should be added or changed, feel free to do so.

Thank you for a great HOWTO.
_________________

... Leaving ground, destination is unknown,
into the darkness and far away from home,
Will your dream come true and what will you find,
when fate is your guide ...

[solarium_rider]

I had a quick question, since nothing is really mentioned (explicetely)on whether it's good or not if the swap file is full. Currently, when i try to view a mpeg from the web or something (through mplayer) the whole system becomes laggy (mouse pointer skips bad.) The desktop also seems laggy when I'm running an emerge sync. I'm using vanilla 2.6.6 kernel. Also is it normal for kdeinit to use 30% of my ram (according to top).

Here's what my stuff looks like:

[Clansman]

what's your swapiness level? do you set it manually?

you could compile a 2.6.7 kernel and include con kolivas' autoregulated swapiness patch in it. i use it.

or, from kerneltrap.org kerneltrap.org/node/view/3366 andrew morton's patchset with nick piggins' addicion could also help you out.

[rich0]

[sanity]

[Rainmaker]

Maybe add this bit:

Your system only reports 28 Megs when you actually have 32 Megs. This is because part of the memory is used for the kernel code, and another part of it is "reserved". Check the ouput of dmesg to see how much this is. The less you "built into" your kernel (i.e. use modules) the smaller your kernel gets.
_________________
If you can't dazzle them with brilliance, baffle them with bullshit.

[jkruer01]

Can someone please explain to me what the benefits of swaping out the contents of cache? The purpose of cache is to store content on a higher speed medium so you can access it faster. If you move the cach to the harddrive then doesn't it defeat the purpose of the cache?

[rich0]

[jkruer01]

Your right, I'm sorry. I didn't take enough time to read every word. Now that I went back and read the original post more slowly it makes much more since to me.

Thanks!

[Stieltje]

I guess that gkrellm does not report free mem the same way that

[sapphirecat]

[Nutterpc]

after having this problem myself.....I always wondered what I could do to try to counteract it

part of it came by reading available values in sysctl -a

part of it came via research......well........most of it really

Now yes, I will admit I used to have issues with linux seemingly runnin off with all my RAM, and then going to swap, but with 1Gb DDR400 DualChannel.........it shouldn't need to

After nearly a weeks worth of research so far, I've come up whit some fairly drastic changes in sysctl.conf, which, on my machine......are producing very noticeable results

Programs take less time to load, they don't hog Proc time (which they used to beforehang )
CPU just blips every now and then, and Disk doesn;t even move...unless I load a program

And for all those wondering what kernel I'm running, its ck-sources latest: 2.6.8-ck3

Nutterpc

PS: if anyone else running this kernel would like to try my sysctl.conf settings out to help me further finetune them, my email addy should be in my profile......just contact me on msn
_________________
If it isn't broke, you ain't tweaked it right
Registered Linux User 353232

[smg]

Mh, I'm just asking me, what's faster for my system, 2*256 DDR-Ram @ 400Mhz working as dual ram modules or making out of this 1GB. Can you recommend me one of this possibilities?

best regards 'ash'
_________________
GnuPG-Key-ID: 0xF8C275D4
Fingerprint: 5B6F 134A 189B A24D 342B 0961 8D4B 0230 F8C2 75D4

[Nutterpc]

all depends really, there's quite a few factors that come into it

a) how much memory do u need
b) how much do u want to spend on memory
c) is the $$ needed to get the extra ram going to make it worthwhile for what *you* do?
_________________
If it isn't broke, you ain't tweaked it right
Registered Linux User 353232

[minaguib]

To add to the excellent post, I've noticed something peculiar about `top`s output that I thought I'd point out.

At the header near the top, `top` says that I'm using no swap: