Computer freeze when RAM near full

mr-simon

Not sure if this belongs here or in desktop, but here goes.

When running close to full memory my machine becomes very unresponsive and will eventually freeze.

I've got 12G installed (3-channel, 4G each) but run multiple VMs under vagrant/virtualbox, along with a desktop environment and firefox, etc.

Swap is enabled, 16G. On my SSD.

I've noticed that it's treacherous running multiple VMs because as my RAM starts to fill up I reach a problem where suddenly everything slows to a crawl and will eventually freeze. It's primarily characterised by issues with moving my mouse/keyboard. My mouse will jump & freeze, with the freezes taking progressively longer. Similarly, when pressing capslock there will be a progressively longer delay before the capslock light toggles on and off.

If I'm not quick enough (the above process only takes about 60sec, tops) then my computer will be completely frozen, and even leaving it overnight to sort itself out doesn't bring about any changes.

I've also noticed the above symptoms when doing other RAM intensive tasks such as compiling something with webkit in it.

Killing the memory-intensive processes (nuking a VM, killing 'emerge') solves the problem and the computer quickly returns to normal.

Swap never gets very full during this process. There's nothing in dmesg that indicates that the kernel has anything to say on the matter.

Any suggestions, please?
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."

eccerr0r · Posted: Tue Jul 05, 2016 2:09 pm Post subject:

More RAM or shrink VM images, adding swap to each individual VM.

Most VMMs will disallow swapping of VM images on the host, it becomes too obvious to the guest OS.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

mr-simon · Posted: Tue Jul 05, 2016 2:12 pm Post subject:

Thanks, but as I mentioned it's not just VMs - any RAM intensive operation will do it.

When running VMs I'm leaving a minimum of 4G headroom for my host OS, so there still should be enough RAM to play with.
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."

eccerr0r · Posted: Tue Jul 05, 2016 2:20 pm Post subject:

Suggestions is still the same. Run 'top' and make sure the sum of all RSS of all VM images isn't adding up to a significant portion of your physical memory.

Merging C++ programs use a lot of ram, especially when building multithreaded. VMs will choke off swap making it hard for your PC to do its job.

So 12GB RAM, two 4GB VM images, is that all? How much ram is free in a quiescent state?

[EDIT]
Just my VM machine. I only have 8GB RAM on my machine, but minimize the VM containers (KVM/QEMU) and the machine runs OK. I had less than 8GB RAM before and kept on running into RAM issues as the VMs grew in size or ran multiple VMs. A LOT of headroom is needed I've found out, but it may be due to KVM/QEMU.

Logicien · Posted: Tue Jul 05, 2016 3:38 pm Post subject:

With VirtualBox it's a good idea to not give more Ram to the virtual machine than the quantity recommanded by VirtualBox.

You can use the sysctl command to tune the kernel variable vm.swappiness . The value of 0 limit the usage of the swap to the strict minimum, higher values allow swapping more.

Swappiness
_________________
Paul

mr-simon · Posted: Tue Jul 05, 2016 3:58 pm Post subject:

Well, here's my world under normal load. Note there's a VMWare VM which has been given 6G to play with, and a VBox VM (headless, under vagrant) that has 2G. This is a normal workload for me.

eccerr0r · Posted: Tue Jul 05, 2016 4:19 pm Post subject:

Yeah you do have a problem with your machine. Changing swappiness won't help.

Something is locking a lot of memory. You only have a bit over 1GB of memory free to work with, and this is NOT enough to do multithreaded C++ builds. Could you sort by resident memory size instead of shared memory? Then again the problem seems to be in the buffer utilization... what else is special about this machine? Special hardware? special (buggy?) filesystems? Kernel memory leaks (what version)? hmm...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

mr-simon · Posted: Tue Jul 05, 2016 4:38 pm Post subject:

By RES:

Logicien · Posted: Tue Jul 05, 2016 11:29 pm Post subject:

I think that swap from memory to hard disk and reverse is not a good idea in itself. It's just a workaround for the lack of Ram. Enough Ram mean no swap. I think that for programs execution, the processor is better to stay on the same bus. It's the reason why vm.swapinness=0 in all my machines.

I prefer the kernel to delete pages in Ram and read again the binary code to load it in memory again. In both cases, swap and not swap implies hard disk access but, less the code move from memory best it is in my opinion.

Your problem can effectively, like tell you eccerr0r, come from a process who is devouring memory and/or the processor time and/or a bus bandwitdth. Maybe you will have better results if you disable swap completely for at least, testing it.

The Linux kernel configuration may be a cause too.
_________________
Paul

eccerr0r · Posted: Tue Jul 05, 2016 11:54 pm Post subject:

It looks like your machine has low uptime, so you have an easy way of testing this:
Try loading applications one by one and see which one jumps your buffer cache up and free memory down.

Perhaps check these numbers on a fresh boot without things loaded.

One possibility that I do not know much about is perhaps something is setting your cgroups to limit the amount of RAM you have. Perhaps a bad configuration somewhere. Again I don't know much about cgroups and how systemd uses them, but perhaps this will jog your memory on some config option that may have been set somewhere by mistake.

(and I'm assuming you're using systemd since it looks like you're using gnome? Or is that a bad assumption?)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

gargoylle_ltk · n00b Joined: 28 Jun 2004 Posts: 33

Try turning swap off as well - I've seen machines under high memory pressure (running MongoDB databases) crawl to a halt when trying to swap but remaining quite responsive without it. The flipside is that the kernel WILL start to kill processes as the machine starts to run out of RAM.

haarp · Guru Joined: 31 Oct 2007 Posts: 535

You're using VMware too? I've had trouble with very sluggish/freezing hosts systems while VMs were running, even below max memory usage. Turns out that hugepaged defrag was screwing things up big time.

you could try

duby2291 · Guru Joined: 17 Oct 2004 Posts: 583

Well, you are obviously swapping. Um, the only thing you can do is add more ram. A lot more ram. At 16GB, I'd increase it to 64GB. You can increase the swappiness value, which just spreads swaps out over a greater distance of time and can make it less terrible. But really the only solution is much more ram.