| View previous topic :: View next topic |
| Author |
Message |
tv007
n00b
Joined: 06 Aug 2006
Posts: 22
|
 Posted: Fri Oct 11, 2013 7:52 pm Post subject: strange memory allocation issues / overcommit |
 |
|
Hi all,
I've done a minor system update two days ago, and since then I struggle with strange "cannot allocate memory" issues. It seems to be tightly related to overcommit but I fail to understand why/how.
I do have a machine with 8GB of RAM, and 4GB of swap that is virtually never used (I've never seen anything being written to it).
Since a minor "emerge -uD world" two days ago, I regularly run into issues with allocating memory. For example I've been unable to emerge some java packages, because javac failed with "Cannot allocate memory". At first I thought it's because of Java being greedy, but then this started to affect other apps. For example after executing "emerge -uD world" and locking the screen, I'm unable to log-in again because the kdm process mysteriously crashes.
After a lot of poking, I've found out that disabling overcommit (i.e. setting /proc/sys/vm/overcommit_memory to 0) the problems suddenly go away. I thought I understand how overcommit works, so either I was wrong or something changed recently. My /etc/sysctl.conf looks like this:
| Code: | net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
kernel.shmmax = 8589934592
kernel.shmall = 2097152
vm.swappiness = 20
vm.overcommit_memory = 2
vm.overcommit_ratio = 75
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10 |
Which should give me ~10GB memory for allocations (swap + 0.75 * RAM). However most of the time 'free -m" shows this:
| Code: | # free -m
total used free shared buffers cached
Mem: 7821 2996 4825 0 207 1717
-/+ buffers/cache: 1071 6750
Swap: 4095 0 4095 |
So there's ~4GB of free RAM, but if I run emerge (consuming a very small amount of RAM) and lock the screen, the kdm will crash on log-in. If I set overcommit_memory to 0, it immediately starts working again.
So, what am I missing?
BTW I'm running kernel 3.11.2 with x86_64 (but I've seen the same issues on 3.10.4 right after the upgrade):
| Code: | | Linux rimmer 3.11.2-gentoo #1 SMP PREEMPT Thu Oct 10 22:41:11 CEST 2013 x86_64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz GenuineIntel GNU/Linux |
|
|
| Back to top |
|
 |
i92guboj
Bodhisattva
Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)
|
| Posted: Sat Oct 12, 2013 1:26 pm Post subject: |
|
|
It's rare...
Did you try rkhunter from a livecd?
The first thing I'd do is to use any live media to download a new kernel from kernel.org, configure and compile it, then install and boot that. Since you have already sorted out the java memory management system, and apps outside that are also giving you trouble, then the only thing that's left is kernel memory management. And, obviating nasty bugs that shouldn't ever happen, the only thing that could cause that is a buggy or malicious kernel module. I guess... |
|
| Back to top |
|
|
tv007
n00b
Joined: 06 Aug 2006
Posts: 22
|
| Posted: Sat Oct 12, 2013 4:56 pm Post subject: |
|
|
| I'll give rkhunter a try, however I'm really wondering how effective it can be on a system with so many possible variations thanks to USE flags and compiler optimizations. I'd guess a misconfiguration is much more likely. |
|
| Back to top |
|
|
tv007
n00b
Joined: 06 Aug 2006
Posts: 22
|
| Posted: Sat Oct 12, 2013 11:39 pm Post subject: |
|
|
I've done some more investigation today, and it seems all this is strongly connected to locking the screen in KDE.
The important metrics here are "CommitLimit" and "Commited_AS" in /proc/meminfo - which shows what is the current amount of memory available for overcommit, and already committed memory. Once "CommitLimit < Commited_AS" it's game over (with overcommit_memory=2). With overcommit_ratio=75 the the CommitLimit is 10201408 kB (i.e. 10GB). Now let's see Committed_AS:
without X
| Code: | | Committed_AS: 181972 kB |
after starting xdm (i.e. a login screen)
| Code: | | Committed_AS: 292848 kB |
after logging in
| Code: | | Committed_AS: 5178620 kB |
Well, obviously a huge jump, for some reason.
running "emerge -uD world -pv" (while still being logged in)
| Code: | | Committed_AS: 5568848 kB |
small increase, nothing terrible
after locking the screen
| Code: | | Committed_AS: 7775460 kB |
and after logging-in again
| Code: | | Committed_AS: 5702276 kB |
So, the amount of memory committed to KDE is pretty huge, for some reason (~5GB). However the really strange thing is that locking the screen increases the amount by 2.5GB. That's something I don't understand.
Anyway, I originally used overcommit_ratio=50, so the CommitLimit was only 8GB. That's pretty close to the 7.7GB. Imagine a few more applications started (Firefox, ...) and things start crashing.
The +2.5GB because also explain why I have missed this at first - when tracking the memory from KDE everything seemed just fine. The only reason I noticed this is by looking at memory from the console, after the KDE login crashed. |
|
| Back to top |
|
|
tv007
n00b
Joined: 06 Aug 2006
Posts: 22
|
| Posted: Sun Oct 13, 2013 2:16 pm Post subject: |
|
|
Seems the KDE hunch was correct. I tried to remove the .kde4 folder (with various KDE settings) and restarted the xdm. And now the numbers look like this:
| Code: | CommitLimit: 10201408 kB
Committed_AS: 1956924 kB |
which is way lower than before. And it does not change if I lock the screen.
So I assume some change in one of the recent KDE versions started to interact with the data in .kde4 a bit differently, which resulted in unexpected behavior of some components - not crashing, but requesting much more memory.
Not sure if it's worth further analysis (what exactly went wrong / how to prevent it from happening again) or how to do that. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
