Gentoo freezes when it runs out of RAM

[mikexx]

Hi,

my laptop is very slow/freezes when it runs out of memory. The laptop has 16GB RAM and 16GB swap (SSD). However, as soon as ca 15 GB (RAM) are used the system freezes, for example moving the mouse pointer or enter the password in the lock screen takes more than one hour. I can observe that when chromium is emerged or when several VMs are open.
I am posting that in the kernel forum as I think it is because of a kernel configuration.
Any ideas?

Best
mike

[1clue]

What do the logs say? Is there anything in any logs concurrent with the frozen state? Or before?

As well, have you definitely linked this to an out-of-memory scenario in more than one instance?

I have a Gentoo box with the same memory, with some amount of swap configured (don't know off hand) but it does not have this problem.

I would suspect that, if it's actually swap-related, there would be some sort of log messages about swap. Or disk access. Or, if it's related to something else you should get messages about that. The fact that you still get extremely slow activity means it's not a kernel panic, so there should be a log message somewhere.

What comes to mind is an extreme thrashing situation. Or deadlocks without a means to break out.

[mikexx]

I didn't see anything suspicious last time. I will check it again.

Best
mike

[mikexx]

I've checked the kernel logs again. Nothing has been logged about swap or something else what could explain the behavior.


Best
mike

[guitou]

Hello.

Most probably a silly question, but did you double check that your swap part is active?

++
Gi)

[P.Kosunen]

All systems freeze when swapping start. You can disable swap to not to freeze, but then some processes die when you run out of memory.

[mikexx]

@guitou
Swap is active.

@P.Kosunen
I am almost sure that I used systems in the past that did not freeze that way.


Best
mike

[1clue]

Re: freezing when swapping starts, it's usually transparent to the user. Operations that normally happen in RAM are stopped, and disk activity happens to page RAM out to the swap device.

If your system has a noticeable lag when swap starts there's a problem, or you have a really slow system in the first place, and a slow disk. If the system gets into a really bad place, swapping out huge amounts of RAM to disk and swapping other huge amounts back, then you can get noticeable lag. If the system is about to fail in spite of the swap, the lag can be severe.

That said, the very premise of swap is simply a last-ditch effort to keep a process working even though memory ran out. It's supposed to be a rare thing, not something that happens all the time. If you frequently have swap activity on your system you will get significant performance hits on long-running processes, and you will also have a higher probability of disk failure.

[PCmaniaK]

My two cents: Do not USE="jumbo-build" emerging chromium. It takes all the memory and asks for more.

[mikexx]

[C5ace]

Run sys-process/iotop in a terminal and see which process reads and writes from and to disk.

[Ant P.]

What disk scheduler are you using? (grep '' /sys/block/*/queue/scheduler)

[Goverp]

I'd say you need to dissect /proc/meminfo. I've been reading up on things like HugePage support, which sounds great, but HugePages are unswappable. (Not saying this has anything to do with this particular situation, just that there be sharks in them there waters. I'm a complete novice here.) There are probably lots of other things that meminfo might be able to warn of.

One thing I found related to emerges - you'll find recommendations to use tmpfs for portage temporary files. And it's great for small apps. BUT if you use it for emerging big ones like Rust, webengine, LLVM, and chromium, it can be a disaster. You end up with the system having to swap out the temporary files to make space for the compilation, which wants to read the temp files and so swaps them back in. In essence, don't overcommit memory - tmpfs should be no more than say 50% of memory. I expect this gets even worse if you use "-pipe" as also recommended. Basically, for the big beasts, use package.env settings to switch OFF all the "go-faster" settings or your machine grinds to a halt.

Unfortunately this sort of problem becomes increasingly common - developers assume vast amounts of memory for cache, big data trees and so forth, and forget they are cohabiting with other apps doing the same. Many years ago I used an IBM mainframe with less than 1MB memory running a multi-user VSE/CICS system under VM, and we had "balanced systems guidelines" that reflected best practice, and explained how to identify memory hogs. These days I doubt even a keyboard poll routine would be rated "good".
_________________
Greybeard

[mikexx]

[tholin]

When you say "my ssd (encrypted)" does that mean you have an encrypted swap partition with dm-crypt? If so that it likely the problem.

Swap on dm-crypt is problematic. I don't know for sure what the problem is but I suspect dm-crypt does memory allocations when data is encrypted. The problem with that is you are already low on memory when data is written to swap. If dm-crypt needs to do memory allocations before writing the data those allocations will force out even more data, which will result in even more memory allocations, which will force out even more data... [repeat]
The result is a long stall.

You can test if that is the case by reformatting the encrypted swap partition as a regular swap partition.

https://bugzilla.kernel.org/show_bug.cgi?id=200105
Here is a bug report I found while googling the problem a while back. There is no final conclusion in that report.

[mikexx]