[RFC/RFT PATCH v3] sched: automated per tty task groups

[tld]

For anyone interested, I've implementing the .bashrc version of this under the current stable gentoo-sources-2.6.35-r12, and in my case the improvement for a lot of things I do is very noticeable. I do a lot of things from urxvt prompts. I also haven't used a login manager in a long time. I log in to a console and use startx. One thing I'm unclear on is whether or not that means that everything I run out of X that isn't via another console uses the cgroup of that original login(??).

In any case, there's a lot of information out there that's confusing as far as setting this up. Here' the setup I'm using. Here are the relevant kernel settings:

Code: Select all

grep CGROUP .config
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_MEM_RES_CTLR=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set

One thing I found very confusing is all the instructions that specify mounting cgroup to /sys/fs/cgroup/cpu which (at least in the kernel I'm running) doesn't exist no matter what kernel options I use. I've seen other instructions specifying /dev/cgroup etc. It apparently doesn't make any difference where you mount that as long as everything involved references the correct mount point. I created a /cgroup directory and used that for everything.

This is in my .bashrc:

Code: Select all

if [ "$PS1" ] ; then  
	mkdir -m 0700 /cgroup/user/$$
	echo 1 > /cgroup/user/$$/notify_on_release
	echo $$ > /cgroup/user/$$/tasks
fi

I've create this script at /usr/local/bin/rmcgroup:

Code: Select all

#!/bin/bash
rmdir /cgroup/$1

...with execute permissions of course.

I've added this to /etc/conf.d/local.start so as to run on every boot:

Code: Select all

# /etc/conf.d/local.start

# This is a good place to load any misc programs
# on startup (use &>/dev/null to hide output)

mount -t cgroup cgroup /cgroup -o cpu,release_agent=/usr/local/bin/rmcgroup
mkdir -m 0777 /cgroup/user

That all works great and the /usr/local/bin/rmcgroup cleans up when all tasks for a group end. Nice stuff.

Tom

[.yankee]

For anyone interested, I've implementing the .bashrc version of this under the current stable gentoo-sources-2.6.35-r12, and in my case the improvement for a lot of things I do is very noticeable. I do a lot of things from urxvt prompts. I also haven't used a login manager in a long time. I log in to a console and use startx. One thing I'm unclear on is whether or not that means that everything I run out of X that isn't via another console uses the cgroup of that original login(??).

You can check that easily looking at the list of PIDs in the tasks file of your session's cgroup (that is, /cgroup/user/[your_console_session_id]/tasks in your case) and comparing it with the list of running processes (be it an output of "ps" or "top" or any X-frontend). Generally, what gets into the cgroup of the session where you've run startx depends on how you run your .xinitrc and whether, inside your .xinitrc, you run your window manager using the exec call.

For instance, I use my own little script that, upon login, checks whether I am under a real tty and if so, presents me with a dialog where I can choose either to start one of my DEs, or just the console. The important part is, this script gets sourced from /etc/profile if I am both non-root and under a real tty - and then, since my .bashrc wouldn't get sourced while stating X in this way, do the cgroup-magic, as in any normal .bashrc.

One thing I found very confusing is all the instructions that specify mounting cgroup to /sys/fs/cgroup/cpu which (at least in the kernel I'm running) doesn't exist no matter what kernel options I use. I've seen other instructions specifying /dev/cgroup etc.

That is because you use a <2.6.36 kernel. the /sys/fs/cgroup mountpoint was introduced later. And the "cpu" sub-directory needs to be created manually (this is optional anyway).

This is in my .bashrc:

Code: Select all

if [ "$PS1" ] ; then  
	mkdir -m 0700 /cgroup/user/$$
	echo 1 > /cgroup/user/$$/notify_on_release
	echo $$ > /cgroup/user/$$/tasks
fi

That might produce a problem in some shells, as did in one of my environments - where you have file overwriting protection enabled for non-root users. In such case using >> instead of > solves the problem.

Code: Select all

mount -t cgroup cgroup /cgroup -o cpu,release_agent=/usr/local/bin/rmcgroup
mkdir -m 0777 /cgroup/user

Oh, that's nice they have a "release_agent" mount option! I didn't know that and used to echo /usr/local/bin/rmcgroup > /sys/fs/cgroup/cpu/release_agent

[Shining Arcanine]

I just applied this, but I did it a little differently than tld. I did a mix of his instructions and the instructions for Ubuntu users.

http://www.webupd8.org/2010/11/alternat ... patch.html

Since it would just be a me too post if I posted instructions, here is a script that you can run as root to do this, provided that you configured your kernel with CGROUPS support in advance (see tld's post):

Code: Select all

#!/bin/sh -- 
# Make the system configure itself for automated per tty task groups before any shells run
cat << END >> /etc/conf.d/local.start || echo "Failure appending to /etc/conf.d/local.start" && exit 1; 

# automated per tty task groups
mkdir -p /dev/cgroup/cpu
mount -t cgroup cgroup /dev/cgroup/cpu -o cpu,release_agent=/usr/local/sbin/cgroup_clean
mkdir -m 0777 /dev/cgroup/cpu/user
END

# Make the shells configure themselves for automated per tty task groups before any shells 
cat << END >> /etc/bash/bashrc || echo "Failure appending to /etc/bash/bashrc" && exit 1; 

# automated per tty task groups
if [ "$PS1" ] ; then
        mkdir -p -m 0700 /dev/cgroup/cpu/user/$$ > /dev/null 2>&1
        echo $$ > /dev/cgroup/cpu/user/$$/tasks
        echo "1" > /dev/cgroup/cpu/user/$$/notify_on_release
fi
END

# Create cleanup script
cat << END > /usr/local/sbin/cgroup_clean || echo "Failure creating /usr/local/sbin/cgroup_clean" && exit 1; 
#!/bin/sh
if [ "$*" != "/user" ]; then
        rmdir /dev/cgroup/cpu/$*
fi
END

# Enable cleanup script to execute
chmod u+x /usr/local/sbin/cgroup_clean || echo "Failure setting execute bit on /usr/local/sbin/cgroup_clean" && exit 1;

# Tell the shell we are finished. It is more elegant this way. :P
exit 0;

Please note that i have not actually tested this, but it should work.

Anyway, I use KDE with compositing. gtkperf times are usually 3 seconds. I was trying to install libreoffice and I noticed some lag. I ran gtkperf and it took 13 seconds. I stopped the libreoffice compile, set this up, restarted KDE, opened konsole and started compiling libreoffice again. It was taking 5 seconds. Then I took some time to type this up and it is now at 16 seconds.

I am not sure if this makes the difference people claim it does. I know that people claim that gtkperf is not a good benchmark, but situations in which I notice lag seem to correspond to situations when gtkperf times are high and situations in which I do not think there is lag seem to correspond to situations when gtkperf times are low. I have PORTAGE_NICENESS=19 in /etc/make.conf, so it could be that the priorities are so low that this tweak does not have any effect. For completeness I would like to mention that I am compiling libreoffice in a tmpfs, although I have 8GB of RAM so there is no swapping being done.

[gringo]

all i can say is that i tried both and i really wasn´t able to see a difference in responsiveness in my slow eeepc, which is where i´m doing all this testing.
The "in-kernel based solution" made it into mainline BTW.

cheers

[Dont Panic]

The "in-kernel based solution" made it into mainline BTW.

Just for clarification, I see it in the 2.6.38_rc kernel, but I don't believe it's in the 2.6.37 kernel.

[gringo]

Just for clarification, I see it in the 2.6.38_rc kernel, but I don't believe it's in the 2.6.37 kernel.

sorry for not being clear : yes, it was pulled in for 2.6.38.

cheers

[jprobichaud]

Reading all this and seeing that all these changes are tty-oriented, I'm wondering: can the cgroups approach be used for keeping users connecting through ssh to a "research box" to stepping on each other toes?

We have a group of machines where users log by ssh (usually, you get "attached" to a pts/0, pts/1, ... if I understood correctly). Sometime, a user launch a script that computes some sufff and almost kills the machine responsiveness for the other users as well. Using 'nice' and even 'ionice' helps a little but not much.

Can "automatic" cgroups policies be defined on a per-user basis? Any examples? I found a "cspeed" tool in the ArchLinux forums, but I have not enough knowledge to see if that's what I'm looking for. The officials admins of these research boxes are very busy. I would like to test these changes on systems I have control on before show this to them.

Thanks!

[devsk]

Reading all this and seeing that all these changes are tty-oriented, I'm wondering: can the cgroups approach be used for keeping users connecting through ssh to a "research box" to stepping on each other toes?

We have a group of machines where users log by ssh (usually, you get "attached" to a pts/0, pts/1, ... if I understood correctly). Sometime, a user launch a script that computes some sufff and almost kills the machine responsiveness for the other users as well. Using 'nice' and even 'ionice' helps a little but not much.

Can "automatic" cgroups policies be defined on a per-user basis? Any examples? I found a "cspeed" tool in the ArchLinux forums, but I have not enough knowledge to see if that's what I'm looking for. The officials admins of these research boxes are very busy. I would like to test these changes on systems I have control on before show this to them.

Thanks!

2.6.38-rc3 autogrouping is based on setsid system call. If a program calls it, its in its own group and can't kill anybody else's performance. So, all you need to do is wrap the shell invoked by sshd with a setsid (part of util-linux-ng) wrapper and create a new session for every user who logs in. This will isolate users from each other.

This solution is very elegant and 2.6.38 makes it possible out of the box...

[Dont Panic]

@jprobichaud

You may want to check out the link kernelOfTruth provided here: http://forums.gentoo.org/viewtopic-p-64 ... ml#6490882

This O'Reilly article explains how to manually create and manage CGROUPS, and may give you some ideas for how to use CGROUPS for more finely grained control than what is provided by the automatic CGROUPS patches.

[kernelOfTruth]

@jprobichaud

additionally you could try to use BFS with "Grouping by UID" support:

see Con's Blog (<-- link inside) for more info

[jprobichaud]

@jprobichaud

additionally you could try to use BFS with "Grouping by UID" support:

see Con's Blog (<-- link inside) for more info

@kernelOfTruth: Thanks for these suggestions (and thanks also "Dont Painic" for pointing me back to the O'Reilly article).

I guess it will already be one big step to move to a new kernel on these machines (right now, they are at 2.6.18, without cgroup support) for the admins, I don't really think they like the idea of getting a "unsupported" kernel patch in This environment is already hard to maintain in good shape (researchers are not necessary computer literate and sometime do all sort of weird/bad things...) we don't want to add too much uncertainties...

I'll digest that. It seems that executing some commands when someone logs in could help keeping everybody in his own little cage, which should help greatly the stability of these nodes. In theory, nobody is supposed to run cpu-intensive tasks there, but in real life, we sometime have these type of trouble...

Again, thanks a lot for these insights!