Limiting CPU and Memory

[Holysword]

Hi there,

So, I'm sure that many have already asked about how to limit a process memory usage. As far as I could find, there is no real solution for this problem.
However, I'm running some few tests now in MATLAB and I need whatever I can have, since with no control at all MATLAB just sucks all the resources and crash my machine.

I've created a simple script which does

Code: Select all

#!/bin/bash
ulimit -Sv $MEM_LIMIT
cpulimit -P /opt/matlab$VERSION/bin/glnxa64/MATLAB -l $CPU_LIMIT &
CPULIM_PID=$!
echo "cpulimit pid is $CPULIM_PID"
echo "Starting Matlab..."
/opt/matlab$VERSION/bin/matlab -desktop -nosplash
echo "Matlab has finished successfully"
echo "Cleaning cpulimit"
kill -9 $CPULIM_PID

First of all, Matlab NEVER finishes by itself and I have to kill it manually for the script to continue, but this is probably a Matlab issue. The second problem is that attempting to set ulimit for any non-root user makes the process to crash prematurely, without any error message, MUCH before any resource is over (I've tried to set values far above those that I actually have). The second problem is that I'm using cpulimit, which... well, its a root-only command. I've tried to symlink it to ~/bin and run it, but I am not sure if that's okay (it doesn't work anyway, because of ulimit).

I don't like the idea of running this as root. Any suggestions?

[eccerr0r]

What's the actual problem here? Is matlab just eating all your RAM?

How much swap do you have? The really crappy performance when OOM happens is typical on 0- or low-swap ("not enough swap") machines. When the computer runs out of RAM it will do executable page swapping which is extremely slow and painful, much worse than having real swap enabled (since only executable pages can be purged, there's not that many executable pages to begin with - most of the memory is used on data pages.) The slowness extends to your X server, and whatever you have running at the same time... making things really bad.

I suspect that matlab may be allocating pages before they're used. With lazy allocation turned on in Linux it will let applications allocate memory before it's used, so to matlab it thinks it has all the RAM it needs.

I think the only real solutions:

- Buy more RAM
- Use a bigger swap file/partition.

[Holysword]

The problem is that I'm using big sparse matrices and attempting to perform ilu decomposition. If they loose sparsity, a matrix which requires 250MB can easily explode to 10GB. I need to tell matlab to stop the run if it hits such a high memory usage.

[eccerr0r]

How much RAM do you have? How much swap?

Is matlab actually allocating the arrays up front (use 'top' while watching it grow)? I no longer have access to a copy of matlab so no idea... I suspect it may be allocating it up front with lazy allocation...

I thought Linux does honor -m (for bash's ulimit), might want to try that too. Then again it looks like it doesn't... hum...

[salahx]

You might want to look into cgroups too. You can limit both memory (see Documentation/cgroups/memory.txt) and CPU usage and other things. Also you don't have to run matlab as root to put it in a cgroup (you may need root access to put it in there, but application itself doesn't need to be root)

[Holysword]

How much RAM do you have? How much swap?

Is matlab actually allocating the arrays up front (use 'top' while watching it grow)? I no longer have access to a copy of matlab so no idea... I suspect it may be allocating it up front with lazy allocation...

I have matrices of 2.5GB. Some of these matrices can run an ILU decomposition nicely without any (significative) growing, while others explode to more than 100 times its original size. This is not a Matlab bug, it is a characteristic of ILU decomposition. But well, I have 4GB + 4GB swap.

I thought Linux does honor -m (for bash's ulimit), might want to try that too. Then again it looks like it doesn't... hum...

Apparentely, in my machine it honors only for root.

[Holysword]

You might want to look into cgroups too. You can limit both memory (see Documentation/cgroups/memory.txt) and CPU usage and other things. Also you don't have to run matlab as root to put it in a cgroup (you may need root access to put it in there, but application itself doesn't need to be root)

Are you talking about this? It sounds interesting, I'll take a look. Any other links you suggest?

[eccerr0r]

It's not a bug per se but it's the OS's (looks like linux too) capability to deal with applications that use a lot of lazy allocated pages. The OS tries to accommodate for applications that do sparse memory allocation, but in this case, it seems to be hurting (or rather, mitigated in userland).

Again we are not talking about CPU usage. The problem is entirely due to swap/memory, limiting CPU usage will not help. I'm not entirely sure why RSS isn't available to the user to limit growth, probably because if there's two competing applications in memory it doesn't make sense... In this case, having 0 swap is fine, then RSS = true allocation.

There should be a number somewhere to tell how many used pages versus allocated but unused pages, I'm surprised there does not appear to be one... weird...

Incidentally, I recall recently on an 8GB machine with 16GB swap (x86-64), I had a process using 16GB and I didn't notice way too much degradation, didn't even realize it was 8GB into swap until I checked... It's all usage pattern of memory I suppose...

[Holysword]

It's not a bug per se but it's the OS's (looks like linux too) capability to deal with applications that use a lot of lazy allocated pages. The OS tries to accommodate for applications that do sparse memory allocation, but in this case, it seems to be hurting (or rather, mitigated in userland).

Again we are not talking about CPU usage. The problem is entirely due to swap/memory, limiting CPU usage will not help. I'm not entirely sure why RSS isn't available to the user to limit growth, probably because if there's two competing applications in memory it doesn't make sense... In this case, having 0 swap is fine, then RSS = true allocation.

There should be a number somewhere to tell how many used pages versus allocated but unused pages, I'm surprised there does not appear to be one... weird...

Incidentally, I recall recently on an 8GB machine with 16GB swap (x86-64), I had a process using 16GB and I didn't notice way too much degradation, didn't even realize it was 8GB into swap until I checked... It's all usage pattern of memory I suppose...

Just remembered that I've got a disk with 640BG that I'm not using. Someone is about to have an epic swap...
In any case, that wouldn't help the responsiveness of the system. I think everybody agrees that the ideal would be to limit both (real) memory and CPU usage, so that the process could be nearly "running in the background".

[eccerr0r]

Again, the problem is not CPU. It's completely memory.

When Linux runs out of RAM and swap (whether you have 0 or gobs of it), it will swap out code pages. Swapped out code pages, especially if it had no choice but to swap out pages that it actually needs to use frequently, will make your computer seem very slow. Just imagine if it had no choice but to discard your keyboard interrupt driver, then every time you pressed a key it will need to refetch from disk... Even worse, mouse driver and video driver... This is why performance tanks when you go OOM.

But as long as you have some swap and Linux does not need to do these drastic measures, it will notice you use the mouse driver and keyboard interrupt driver frequently and opt to swap out a portion of your array that it's not working on at the moment to disk. I don't think you'll notice performance to be that bad... do you notice much of degradation when you're just 2GB into swap?

You should try the swap for the heck of it. I have a machine with 384MB RAM and 1.5GB swap to build firefox9. The machine swaps like mad during build, but the machine was still responsive despite being under extremely heavy swap pressure.

At least until someone knows how to get the kernel to auto-kill using RSS information! (or you could write a script to constantly poll RSS size with ps -F and kill matlab if it gets too big...)

[Holysword]

Again, the problem is not CPU. It's completely memory.

ILU decomposition have high cpu usage, so I still prefer to limit cpu usage as well.

do you notice much of degradation when you're just 2GB into swap?

Yes. I start losing responsiveness when I'm reaching full real memory usage. I don't think swap is as marvellous as you suppose. Plus, now I've got 640GB of swap, and I still lose responsiveness.

[eccerr0r]

Here's one thing about CPU utilization: Doing emerge builds on Gentoo. It also can use 100% CPU, but the machine is much more responsive than when running an application that chews up all memory. That is, if you're also not running out of RAM during builds.

And I'm not saying you won't lose responsiveness, swap will still put heavy weight on the disk backing store and if you need to do demand fetches, swap takes priority over reading new data from disk. If you have a slow disk this will hurt more of course (I sure hope you're not using a USB disk for swap).

If you're just going to let it sit there and crunch and occasionally touch it, yes the user interface code will still get swapped out just like the OOM situation - the unused UI code is wasting memory that it could be using for matlab, and yes it will take a few moments to swap back in. Once it's back in however, it will be usable once more. Keep in mind every moment you aren't using the console will make Linux think you're away and it will once more mark the UI as candidate for swapping out.

I don't "suppose" swap is good, it IS good. It's nowhere near as good as real RAM but it does prevent the "last resort" code page swapping situation that REALLY hurts performance. Not to mention it gives some breathing space to start new applications such as shells to kill offending processes.

Did you at least try the watchdog "solution" to write a script to monitor RSS of matlab, and kill it if it gets too big? This will not work effectively if you're out of memory, so having enough swap is good for this too. Also it looks cpulimit will limit child processes, so you could cpulimit a "su" to a normal user. But this will not affect responsiveness except on 386 or really slow CPU that has a hard time dealing with only the user interface code.

[Holysword]

eccerr0r, I am sorry but I don't need more lessons on operational systems working, I've finished my graduation quite a long time ago, but thanks. I know what a swap is for, I am just TELLING you that it does not prevent the system from losing responsiveness - and I know why. Thanks for explaining anyway.

So, back to where I've mentioned, the ideal would be to tell the kernel to not allocate more real memory for a given process, but only allow it to take from swap. Is that even possible with current kernels? If so, where should I start reading for allowing that?

[salahx]

Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).

Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

[Holysword]

Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).

Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

Thank you salahx, I'll study that this weekend. I remember having read something regarding stability and throughput regarding cgroups. Is it stable for you?

[smartass]

As for sparse matrices, I've heard that NumPy handles them more efficiently....

IIRC, if matlab just allocates the memory space, VIRT will go up, but it shouldn't actually load it into physical RAM until that memory segment is used. And it would unload it from RAM as soon as it isn't being used again if there's not much space left in RAM.

So, to me it sounds like your swap could be corrupt. Performing a HDD health test wouldn't hurt anything, would it?

[Holysword]

Yes, cgroups. Here's Fedora's guide on cgroups, nad in particular, for memory cgroup you can set (eboth hard of soft limits) on amount or ram , max total ram+swap, swapiness, (though not overcommit, there was a patch years ago to that but its was never merged due to lack of a use case). Note that this is TOTAL ram - not just the RSS. THe OOM killer is also per-cgroup (by default, its on).

Current version of openrc even create a cgroup hierarchy for each controller by default in /sys/fs/cgroup. So just "mkdir /sys/fs/cgroup/mem/matlab" and set the appropriate values in the new cgroup. Then move your shell in to the new cgroup (need to be root to move your shell) and run matlab.

I started reading some things, and I found this:
http://forums.gentoo.org/viewtopic-t-86 ... art-0.html

I am actually using jackd as well, and as so I need realtime priority for it. Cgroup + CFS are good enough for realtime applications? (Currently I'm using BFS)

[Holysword]

As for sparse matrices, I've heard that NumPy handles them more efficiently....

IIRC, if matlab just allocates the memory space, VIRT will go up, but it shouldn't actually load it into physical RAM until that memory segment is used. And it would unload it from RAM as soon as it isn't being used again if there's not much space left in RAM.

So, to me it sounds like your swap could be corrupt. Performing a HDD health test wouldn't hurt anything, would it?

Unfortunately in my field I'm obligated to use Matlab for comparison reason.
My swap is the second hard driver of my laptop, which I forgot when I installed my gentoo on it (so used to do everything on /dev/sda that I didn't realise there was a /dev/sdb) and was never used until now. I ran the test in any case:

Code: Select all

◢ perjanus ◣ Lago $  smartctl -H -d ata /dev/sdb
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-zen+] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

◢ perjanus ◣ Lago $  smartctl -l error /dev/sdb
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-zen+] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

◢ perjanus ◣ Lago $  

[smartass]

Oh, sorry to hear that ... just curious, what field is this ? I fool around in the filed of nuclear fusion and even the JET lab switched to NumPy / SciPy.

Well, if your hardware is ok, that gives you at least a solid base to build on.

So, another idea: you seem to be using a fairly recent kernel + zen patches (or is that just your naming?). I'm not saying that swap support is broken there.... but you never know.

I suggest: try running it on the oldest possible kernel that your Matlab version states in their "Requirements" as the minimum. Or just any backup kernel you still have...

Also, your sig shows that you are using the BFQ I/O scheduler. Not saying it's broken or anything, but it may not have been optimized for such huge memory mappings.
So, if older kernels don't help, try cutting down on kernel patchsets which aren't considered "production-quality" yet, especially if they have something to do with I/O.

[Holysword]

Oh, sorry to hear that ... just curious, what field is this ? I fool around in the filed of nuclear fusion and even the JET lab switched to NumPy / SciPy.

Iterative solvers in general. I am using (or trying to) ILU(1e-3) as preconditioner since it seems to be the "default" for most of the test cases in most of the papers I've read. I am using test cases from University of Florida Collection, which includes problems from fluid mechanics, quantum chromodynamics, meteorology and etc. The tests are being run in Matlab because the methods I'm comparing with were published with tests coming from Matlab. I actually managed to obtain the very same results as them with their methods for the cases in which I've got already the ILU decomposition.

Well, if your hardware is ok, that gives you at least a solid base to build on.

So, another idea: you seem to be using a fairly recent kernel + zen patches (or is that just your naming?). I'm not saying that swap support is broken there.... but you never know.

I suggest: try running it on the oldest possible kernel that your Matlab version states in their "Requirements" as the minimum. Or just any backup kernel you still have...

Also, your sig shows that you are using the BFQ I/O scheduler. Not saying it's broken or anything, but it may not have been optimized for such huge memory mappings.
So, if older kernels don't help, try cutting down on kernel patchsets which aren't considered "production-quality" yet, especially if they have something to do with I/O.

I am suspecting of BFS+BFQ since it is meant for regular desktops, and regular desktops do not run sparse matrices computations. Perhaps what I am seeing is an intended behaviour of BFS. Since I have to move to CFS anyway to test cgroups, I will give it a try with pure CFS.

[smartass]

Being able to reproduce some experiment/test data is a great feeling, isn't it ? Here the results change each day...

Yeah, I think the more vanilla sources you use, the better. Gentoo-sources should be fine though.

[Holysword]

I will give it a try with pure CFS.

So, I've switched to pure CFS with 3.2.5-gentoo sources. The responsiveness seems to have increased considerably by doing so, but I think that I was having 2 problems at once. Apparentely there was a memory leak in one of the base libraries of e17 (read this post). Summarizing, CFS + updated e17 solved my problem.

Remarking here for anyone having similar issues: BFS performs really badly for non-canonical situations, like running heavy Matlab computations. In my system, it seems that the kernel was giving all the resources to Matlab, hoping it would finish sooner, resulting in frozen mouse and keyboard and pretty much everything. Even though I knew that it was Matlab's fault, I would have to sit there waiting, since I couldn't "CTRL+C" it or kill the process. CFS seems much more responsible in this sense: at first it gives a lot of resources to Matlab, and I even lose control of mouse and keyboard as well. However, after few seconds the kernel realises that "this sh** is never going to finish" and I regain control of keyboard and mouse, while Matlab runs in a backward-ish behaviour. Matlab takes more time to finish and there is normally a significative slowdown, but not a complete loss of responsiveness - that is, you can actually stop Matlab if you want.

Leaving it "unsolved" until I test the cgroups thing properly though; that would be interesting!

[smartass]

Glad to hear things have improved Thanks for sharing this invaluable experience with BFS, I didn't realize CFS was so smart.