System freeze on low memory

haarp · Guru Joined: 31 Oct 2007 Posts: 535

I know I'm not the first one to report this. Google finds a million articles about the subject, but none offer an actual solution. Maybe someone here knows something.

My machine has 24GB of RAM and twice that for swap. With this I can happily use memory-intensive applications and keep /tmp and other caches as tmpfs (which I assume just counts towards "used" in htop and friends). The system will barely touch the swap for some reason but works well.

Until I run out of RAM that is. Which can happen when running VMs or opening huge images in Gimp. When this happens, the system freezes. Badly.

Facts:

- System slows down to the point that not even the mouse pointer or panel clock will work. It will occasionally twitch and react to input, but only after minutes delay
- Very heavy HDD thrashing visible on the HDD led. Presumably endlessly paging back and forth code pages, which explains why code execution slows to a crawl
- Calling the OOM killer (SysRq-f) does nothing. It dutifully logs the invokement in the kernel log, but doesn't bother actually killing anything
- After a few minutes to hours, the system thaws. It carries on like nothing happened, other than now having a few GB in swap instead of physical RAM

This has happened about a dozen times now. OOM killer doesn't feel like getting invoked and swap only really gets used after a catastrophic freeze has occured. What can I do to make it actually use swap before everything goes to shit? vm.swappiness did not do the trick. I'd also like for the OOM killer to actually do something during a freeze condition.

I have the suspicion that my heavy use of tmpfs may contribute to the kernel not realizing it's running low, but this is only a theory. In any case, it should swap out tmpfs first...

I'm open for any ideas. Thanks!

NeddySeagoon · Posted: Sun Jun 17, 2018 10:26 pm Post subject:

haarp,

Gimp does its own thing for swapping out image data. By default, its in /home/<username>/.config/GIMP/<ver>/
That's likely to be on your HDD somewhere.
Look under Edit/Preferences/Folders, which is at the bottom of the Preferences menu, just out of sight.

For GIMP to do its own thing, it would have to ask for some RAM and lock it, so the kernel would not try to swap it.

Over allocating RAM to VMs is a killer. The kernel is quite happy to swap out the dynamically allocated RAM in use by VMs.
I did manage to allocate 16G RAM to an assortment of VMs on a system with only 8G of physical RAM.
Its a lesson I won't forget.

Swap is only used for dynamically allocated RAM. Anything that has a permanent home on HDD can be dropped and reloaded.
It may need to be written first. Dirty buffers are never written to swap.
This dropping and reloading is a form of swapping
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

benchaney · n00b Joined: 28 Dec 2017 Posts: 30

I think you have understood the problem backwards. The huge freeze you are describing is most likely caused by swap. At some point your system runs out of physical memory and decides to move some pages to swap. The system is frozen because writing multiple gigabytes to disk takes time, and because nothing can be accomplished when the system cannot allocate memory. The solution you are looking for is a way to prevent your system from swapping rather than encouraging it to.

haarp · Guru Joined: 31 Oct 2007 Posts: 535

Neddy,

I know about Gimp's own swap space. The problem is more due to the fact that Gimp leaks memory like an oil well. Opening and closing images barely ever returns the memory that was used by them.

In any case, there is still enough used memory from other apps and tmpfs that can be swapped out. It's just not being done until a freeze, for some reason.

NeddySeagoon · Posted: Mon Jun 18, 2018 9:04 am Post subject:

haarp,

There is no fix for a memory leak other than fix the code.
A workaround is to restart the app.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

haarp · Guru Joined: 31 Oct 2007 Posts: 535

Neddy,

Indeed. But I'm not in the market for fixed apps. I'm in the market for kernels that use swap and execute the OOM killer when necessary.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

-> https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

haarp · Guru Joined: 31 Oct 2007 Posts: 535

This refers to hard oom condition.

From the same document:

NeddySeagoon · Posted: Mon Jun 18, 2018 12:04 pm Post subject:

haarp,

Have you tested that?
Put /tmp on the HDD and see if the problem goes away.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

tholin · Apprentice Joined: 04 Oct 2008 Posts: 203

As you have noticed you are not alone with problems like these.

P.Kosunen · Guru Joined: 21 Nov 2005 Posts: 309 Location: Finland

Edit: Nevermind, did not read all.

bunder · Bodhisattva Joined: 10 Apr 2004 Posts: 5934

would increasing min_free_kbytes help at all? by default its pretty low actually, something like 11mb. i'm experimenting with bumping it up to 64mb myself.
_________________

eccerr0r · Posted: Mon Jun 18, 2018 2:17 pm Post subject:

I think the OOM killer is getting confused or not realizing the extent of the problem.

To the OOM killer, you still have memory available. There's swap available. You haven't completely exhausted all memory but have gotten to the point the system is unusable.

However there seems to be an issue with tmpfs: I've _never_ seen tmpfs get swapped out, as if it is locked in memory much like a VM. I don't know if it's a bug or not but it's stuck in memory, even if it hasn't been recently used.

So the "fix" is probably to depend on VFS cache and not tmpfs (i.e. don't use tmpfs!!!) if you're working on large files and expect to overrun RAM when accounting for both tmpfs usage and regular memory usage.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

haarp · Guru Joined: 31 Oct 2007 Posts: 535

eccerr0r · Posted: Mon Jun 18, 2018 6:41 pm Post subject:

Well the problem is that the biggest consumer of memory may not be gimp or firefox, it may be tmpfs -- which it can't "kill"!

I think there is sort of a "cultural" problem with people trying to outsmart cache by using tmpfs with the historical "ramdisk" methodology, trying to squeeze the last bit of speed. Indeed there are situations where there is a win by using tmpfs -- that is, if the files there are really temporary. But people copying work files into tmpfs to "work" on it is a flawed methodology on systems that share memory between the two.

I know that VMs should not swap out pages else it will become painfully obvious in a VM. Also shared memory should not be swapped out either, there's some race conditions to deal with. Tmpfs is commonly used for shared memory, so I can see why there's a policy not to swap it out...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

haarp · Guru Joined: 31 Oct 2007 Posts: 535

tholin · Apprentice Joined: 04 Oct 2008 Posts: 203

eccerr0r · Posted: Wed Jun 20, 2018 2:43 am Post subject:

I've found that gentoo-sources-4.14.11-r2 actually seems to work as expected, at least if I specify a tmpfs that exceeds physical memory, it does allocate space in swap, so at least this portion is working as expected...

However, gentoo-sources-4.9.95 and 4.12.12 -- it wedges my other machine if I try to allocate a file in tmpfs that exceeds available physical memory.

Different machines however, so may not be comparing the same thing.

If anyone wants to try: first make sure you don't have anything useful running in case your machine dies.
Make sure you have plenty of swap for this experiment. Linux has no options but to page text if it has no swap which kills.

If you have a consolekit or systemd-logind machine, /tmp and /run/user/${UID} are both writeable by the user. If "lucky" both are half the machine's RAM, you just need to create files that sum up to your physical memory. Create a file that fills both of them:

$ cat /dev/zero > /tmp/garbage1
$ cat /dev/zero > /run/user/${UID}/garbage1

If your machine starts getting wedged (mouse stops moving, can't type, etc., etc.) when the second one is running, you have a defective kernel... The expected result should be programs reporting disk full not system failure!

I don't know if control groups or memory limits will prevent a user from letting the machine crash when filling up RAM/tmpfs.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

haarp · Guru Joined: 31 Oct 2007 Posts: 535

Just a small update on this.

@tholin, yes, this is likely what made the OOM killer ineffective. In any case, it wasn't automatically triggered to begin with, which is the main issue. Or was.

I upgraded from 24GB to 32GB RAM. Whoever said that throwing hardware at problems never solved anything? For now I'm saved from it.

I discovered oomd, which Facebook just open-sourced. Might be just what I need should the issue ever return.

https://lwn.net/Articles/760461/