Help interprete RAM usage ("free" and "ps" output)

[222697]

Hi,

this is a generic linux question, not gentoo specific and not hardware related- sorry for that, but since Gentoo is an expert distribution I hope that some geek can help me out with this strange thing.

It's about a Linux hardware box (no VM) with 32 GB RAM. Here the output of the free and ps command (ps output shortened).

How can it be, that free states only 846 MB used, but ps tells that informix (a database software) process uses 14 GB RAM RSS (42%)?

According to the ps man page, the VSZ and RSS values are in KiB.

[nemectic]

I expect the database itself is taking up cache, which is represented by free but still available for other apps to use if necessary.

[222697]

[nemectic]

Yeah, the database can be read fine from disk, but it's obviously much faster from RAM. The process itself will take up an amount of memory, but much of that 42% will be the database which is just cached. If other programs need to make use of the memory some of the cache will be lost and the database will read from disk instead, slowing down access considerably - the process itself will still consume its own allotment of memory though, which will probably be a couple of percent at most I'd guess.

[Atom2]

I would think the answer to your question lies in the mmap(2) system-call which allows a process to memory-map a (regular or device) file to its address space thus showing up in the resident set size (RSS) column in the ps command.

Accessing data being stored in a memory-mapped file from within the mapping process would usually still go through the kernel's file caching algorithms: In case the requested data has been read from or written to recently, requests can easily be served directly from memory through the in-kernel page cache. Otherweise the first access will populate (i.e. fill-up) the page cache to then serve the request.

In case the kernel requires memory previously allocated to the page cache for other purposes (e.g. to start a new process) it will either simply re-use non-dirty cache pages (i.e. those that contain unchanged data only) or, lacking the availability of any such pages, flush dirty pages (those that contain changed data) to disk followed by marking those pages non-dirty thus making those available for re-use.

If you think this through, it would also mean that the RSS of the process will not shrink if the kernel's page cache is re-used elsewhere (albeit, the file is still memory-mapped to the process, though it's data may no longer be in-memory). It would however make access to data in the memory-mapped file slower due to the need to re-populate the requested data to the kernel's page-cache.

As my system is currently down, I can't test this at the moment, but

[Naib]