[RFC/RFT PATCH v3] sched: automated per tty task groups

[albright]

it's over-hyped unless you're doing a lot of Konsole/Terminal stuff concurrently

I have snapback doing backups every hour which makes my desktop
"stutter" or "pause", and snapback uses ttys (I would think). So I'm thinking
this might help.

[SlashBeast]

I wonder if I can mount no only cpu there but also something to control IO. Anyway I did it the-shellscript-way a little different, If anyone care:

Code: Select all

# cat /root/cgroups-prepare.sh 
#!/bin/sh

mkdir /dev/cgroup -m 0755
mkdir /dev/cgroup/cpu -m 0755
mount cgroupCpu -t cgroup -o cpu,release_agent="/root/cgroups-wipe.sh" /dev/cgroup/cpu
mkdir -m 0777 /dev/cgroup/cpu/user

Code: Select all

# cat /root/cgroups-wipe.sh 
#!/bin/sh
for cgroup in $*; do
	rmdir /dev/cgroup/cpu/$cgroup;
done

and my zshrc:

Code: Select all

# cgroups magic. Doing what does 200 lines miracle patch.
if [[ "$-" =~ "i" ]] && [ -d /dev/cgroup/cpu/user ] && [ -w /dev/cgroup/cpu/user ]; then
	mkdir -m 0700 /dev/cgroup/cpu/user/$$
	echo $$ > /dev/cgroup/cpu/user/$$/tasks
	echo "1" > /dev/cgroup/cpu/user/$$/notify_on_release
fi

[Dont Panic]

I'm always dubious about patches like this one, especially when Phoronix starts shouting from the tops of the towers how fantastic the feature is.

But I've done a fair amount of benchmarking, and there seems to be a lot of places where task group scheduling helps (sometimes very noticeably), and I couldn't find much downside in my testing.

My tests used a kernel compilation with -j6 on a single CPU system (AMD Athlon(tm) XP 2600+) in conjunction with the latency test wakeup-latency cited in Mike Galbraith's LKML posts discussing the patch (http://lkml.org/lkml/2010/10/19/212). In one console I would make a kernel after a fresh reboot and a 'make clean' using: 'time make -j6 && killall wakeup-latency'. In another window, I ran the wakeup-latency benchmark, which was terminated at the end of the kernel compilation by the 'killall' command.

Please keep in mind that even though the wakeup-latency test shows eye-popping numbers, it is a purpose-built benchmark to test latency, and these results don't always translate into real-life improvement. Having said that, I do notice an improvement in Xorg performance (in terms of latency) when running CGROUPS.

I also wanted to compare a kernel with the "sched: automated per tty task groups" patch versus Lennart Poettering's manual method (See Fitzcarraldo's post above) versus Group Scheduling disabled versus Group Scheduling not built (I'd heard that even having Group Scheduling would degrade performance).

My conclusion was that latency could be improved considerably by implementing CGROUPs scheduling without impacting system performance as measured by a kernel compilation (as with anything like this, I'm sure there are performance regressions out there somewhere).

There's certainly considerable variation in the wakeup-latency results, but I was more looking at when it made changes of two orders of magnitude.

I wish I knew how to do tables on this forum, but here's my results

Test #1 (baseline): average latency: 2592.5 µs Kernel Build Time: real 21m13.003s user 18m49.836s sys 1m32.097s
Test #2: average latency: 8.1 µs Kernel Build Time: real 21m13.089s user 18m53.186s sys 1m30.387s
Test #3: average latency: 32.0 µs Kernel Build Time: real 21m20.134s user 18m25.428s sys 1m49.036s
Test #4: average latency: 5600.8 µs Kernel Build Time: real 20m58.894s user 18m39.197s sys 1m31.424s
Test #5: average latency: 15.7 µs Kernel Build Time: real 21m1.724s user 18m45.550s sys 1m29.031s
Test #6: average latency: 6595.3 µs real Kernel Build Time: 20m45.245s user 18m34.281s sys 1m25.808s

Trial Descriptions:
Trial #1: Baseline trial. I used a 2.6.35.7 kernel without any "sched: automated per tty task groups" patch or manual task group scheduling. CGROUP task scheduling was built into the kernel (which seems to be the default usually, even with mrproper), but not implemented. Ran from console in TTY1 and TTY2.
Trial #2: I used a 2.6.36 kernel patched with the "sched: automated per tty task groups V3" patch, and allowed automated task groups to schedule the tasks. Ran from the console in TTY1 and TTY2.
Trial #3: A repeat of trial #2 with a patched 2.6.36 kernel, but I ran it under X11 (using IceWM, a fairly lightweight wm). Since I started X on TTY1 in this trial, I was curious to see if performance was affected by effectively running all tasks under the same TTY (the patch normally separates task groups based on TTY I believe).
Trial #4: Essentially a repeat of trial #2, but booted the kernel with the "noautogroup" boot parameter. The purpose of this trial was just to confirm that the "noautogroup" boot parameter really does turn off automatic group scheduling since I want to test the manual method in the next trial. Ran in console on TTY1 and TTY2.
Trial #5: A repeat of trial 4 with the "noautogroup" boot parameter on a patched 2.6.36 kernel. However, I implemented manual task group scheduling using Lennart Poettering's manual method (See Fitzcarraldo's post above for links). Tasks were run in console on TTY1 and TTY2, however, both tasks were grouped in the same category.
Trial #6: Essentially a repeat of trial #1, except this was a 2.6.36 kernel built with CGROUPs totally disabled. I had read that merely having CGROUPs enabled in your kernel incurred a performance penalty. However, the difference was probably within the noise level.

Even though I haven't performed a statistically rigorous number of tests, the kernel compilation times are fairly tightly grouped, so I've concluded that having CGROUPs will not significantly hurt or help things like compiling. The results of the wakeup-latency benchmark seem to vary. I'm not sure you'll notice the difference between an average latency of 25 and an average latency of 10, but when the latency changes by two orders of magnitude from 25 to 2500, this is a change I can feel when running X11.

But it's important to qualify the wakeup-latency result in that you're not really running 100X better with group scheduling. But you can now officially count me in the group that concludes there's a noticeable improvement running some implementation of group scheduling without much downside that I could identify.

[Fitzcarraldo]

Thank you for putting in all that effort to benchmark the different kernels and userspace approaches, Dont Panic. Very interesting and helpful.

I notice from LKML that the patch is still evolving. The latest iteration, Version 4, has moved away from using task groups per TTY to using task groups per session (http://lkml.org/lkml/2010/11/20/91). I think this new approach could be more relevant to 'normal' desktop users than the per-TTY approach.

[Shining Arcanine]

Thank you for putting in all that effort to benchmark the different kernels and userspace approaches, Dont Panic. Very interesting and helpful.

I notice from LKML that the patch is still evolving. The latest iteration, Version 4, has moved away from using task groups per TTY to using task groups per session (http://lkml.org/lkml/2010/11/20/91). I think this new approach could be more relevant to 'normal' desktop users than the per-TTY approach.

Here is a backport for 2.6.35.8:

http://paste.pocoo.org/show/294282/

[asturm]

Am I the only one experiencing drawbacks of that patch? Something as simple as emerge calculating dependencies (utilizing only one of my two cores) brings sound output (plain ALSA here) with Amarok2 to its knees (stuttering playback).

I am using the version from Gentoo patchset v4 (the one in masked gentoo-sources-2.6.36-r2) against tuxonice-sources-2.6.36 running on a decent enough dual-core system (X200s). My system might be exotic in that it is equipped with a Seagate Momentus XT 500GB HDD including 4GB SLC SSD cache, with the portage tree quite likely residing inside the cache due to its regular use (and emerge --sync really is pretty fast).

[kernelOfTruth]

Am I the only one experiencing drawbacks of that patch? Something as simple as emerge calculating dependencies (utilizing only one of my two cores) brings sound output (plain ALSA here) with Amarok2 to its knees (stuttering playback).

I am using the version from Gentoo patchset v4 (the one in masked gentoo-sources-2.6.36-r2) against tuxonice-sources-2.6.36 running on a decent enough dual-core system (X200s). My system might be exotic in that it is equipped with a Seagate Momentus XT 500GB HDD including 4GB SLC SSD cache, with the portage tree quite likely residing inside the cache due to its regular use (and emerge --sync really is pretty fast).

yeah,

I had the previous version of that patch (v3) and my desktop's responsibility - especially the mouse's movement - was pretty jerky while compiling chromium with -j9 (core i7 860 with reniced portage to 10) so I've completely disabled cgroups again

now it works most of the time fine (I have a heavy patched 2.6.36 kernel so it might be much better compared to vanilla 2.6.36 but it's enough for me )

[asturm]

No issues with mouse yet. However, I can also reproduce the problem with kaffeine - output stops reliably when firing off an 'emerge -uvaDN world'.

[kernelOfTruth]

No issues with mouse yet. However, I can also reproduce the problem with kaffeine - output stops reliably when firing off an 'emerge -uvaDN world'.

this might be more related to

dm-crypt, i/o-scheduler, filesystem, mount options, VM in general - especially vmscan, etc.

[asturm]

Btw, mouse and system just completely hung for several seconds during kmplayer playing an Xvid movie until the 'emerge -uvaDN world' had finished. I definitely didn't have those issues without the patch.

I am operating a fairly basic system without fancy stuff, no crypt, only regular ext4 partitions +mounting options (only noatime), as for IOSCHED:

Code: Select all

CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_IOSCHED="cfq"

EDIT: Going to try out the effect of CONFIG_IO_DELAY_NONE=y

[Shining Arcanine]

Btw, mouse and system just completely hung for several seconds during kmplayer playing an Xvid movie until the 'emerge -uvaDN world' had finished. I definitely didn't have those issues without the patch.

I am operating a fairly basic system without fancy stuff, no crypt, only regular ext4 partitions +mounting options (only noatime), as for IOSCHED:

Code: Select all

CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_IOSCHED="cfq"

EDIT: Going to try out the effect of CONFIG_IO_DELAY_NONE=y

I have had these problems too. :/

I have compositing on in KDE and I am using Nvidia's binary driver.

[asturm]

KDE-4.5.3 and xf86-video-intel-9999, already tried with and without compositing - with the same result. Any things that could stand out on your system compared to the average setup?

@EDIT: CONFIG_IO_DELAY_NONE=y made no difference whatsoever.

@EDIT #2: ...as did CONFIG_DEFAULT_IOSCHED="noop"

[Yamakuzure]

As I am not doing any multimedia stuff on my companies laptop, I gave the "alternative" (like described by SlashBeast and Fitzcarraldo) a try.

I have to do a lot of stuff inside a Virtual Machine (Windows XP-32, VMware Workstation 7, 768MB Ram from my laptops 2GB) and normally KDE4 apps, like thunderbird and firefox are quite unresponsive when the VM is running. (Even when Windows idles.)

With the "alternative" patch I started VMware Workstation via Konsole, thus with cgroups support, and while I am typing this everything is quite responsive. In the meantime Windows has installed Microsoft Office 2007 Service Pack 1 (268MB) and I had no issues with it. The only hint that the VM is working is the noise from the CPU fan and the hard drive.

So if this kernel patch (not the tty-only alternative) is going into the kernel for genral scheduling rather than tty-only, I think it's a more or less huge leap.
Well... at least when the drawbacks are sorted out...

[.yankee]

Am I the only one experiencing drawbacks of that patch? Something as simple as emerge calculating dependencies (utilizing only one of my two cores) brings sound output (plain ALSA here) with Amarok2 to its knees (stuttering playback).

I am using the version from Gentoo patchset v4 (the one in masked gentoo-sources-2.6.36-r2) against tuxonice-sources-2.6.36 running on a decent enough dual-core system (X200s).

Are you using the in-kernel patch, or the userspace implementation? In the latter case, how do you start your X? Did you assure it gets its own cgroup?

[devsk]

The kernel patch has run into some issues. There is more discussion around it on lkml. Looks like more changes are coming in sched. Looks like the final patch may not be safe for anything other than 2.6.38.

https://lkml.org/lkml/2010/11/28/47

[.yankee]

The patch works for me without noticeable drawbacks - I used the one posted on 24.11.10 against vanilla-sources-2.6.37_rc3 (for autogrouping) as well as userspace cgroups managing implementation (modifying .bashrc and .xinitrc). With these, I haven't noticed the downsides genstorm described.

[gringo]

The kernel patch has run into some issues.

i noticed several problems with v4 of this patch ( processes get stuck and 2 kernel panics, one of them maybe unrelated), v3 works actually way more better for me.

cheers

[Dont Panic]

I see Mike Galbraith posted a v4+ patch on Tue, 30 Nov 2010.

https://lkml.org/lkml/2010/11/30/121

The notes indicate this patch is against "tip". Is the linux "tip" branch 2.6.38, or still with the 2.6.37_rc branch?

Has anybody tried this version yet against a 2.6.36 or 2.6.35 kernel?

There was some discussion about this patch addressing some issues strictly in the "tip" branch, so I was wondering if it was worthwhile to backport this patch.

[.yankee]

Has anybody tried this version yet against a 2.6.36 or 2.6.35 kernel?

From what I've read on the kernel mailing lists earlier, the whole autogroup patch depends on another one, which got into .37 development. So unless you have a patched version of .36, it won't work (won't patch).

[Yamakuzure]

The kernel patch has run into some issues.

i noticed several problems with v4 of this patch ( processes get stuck and 2 kernel panics, one of them maybe unrelated), v3 works actually way more better for me.

cheers

Not only the kernel patch. I have changed my "alternative" to a script to start the cgroups stuff manually instead of automatically starting it with /etc/conf.d/local. The reason is, that I had a BSOD with a Kernel oops (something about a failed page request), and after that VMWare WS didn't start any more. It didn't print an error, write something into a log, nothing. It simply didn't do anything anymore. After I took the cgroups stuff out of autostart and rebooted, it worked again. And this was the weirdest BSOD I've ever seen. After five seconds or so, my mouse cursor reappeared, and when I clicked the left mouse button the BSOD disappeared and my desktop was back, like nothing happened. Strange. I first thought it had to do with the ITCO watchdog in this laptop, but both watchdog and hangcheck would have rebooted the machine, wouldn't they?

[asturm]

Are you using the in-kernel patch, or the userspace implementation? In the latter case, how do you start your X? Did you assure it gets its own cgroup?

In-kernel ofc - but not anymore.

[.yankee]

In-kernel ofc - but not anymore.

Well, I'm running a patched kernel right now and going through python-updater rebuilding 100 packages with -j10 emerge option and -j4 in MAKEOPTS and the only slight glithces I've noticed so far (93 of 100 complete now) were when the loadavg went up to

Code: Select all

13.2, 9.5, 7.0

- but they were really slight, nothing I would not expect when running such emerge under an unpatched kernel.

Which version of the patch have you used? This might narrow down where the problem occurs...

[DaggyStyle]

I've heard that there are performance hits when system isn't in stress, did anyone noticed it?

[wrc1944]

I just booted the new OpenSuse kde-4.6 beta live cd (milestone 4, 32 bit version) because I read it had adopted this patch.
FWIW, here's the cgroup config section from their default boot kernel "Linux/i386 2.6.37-rc3-git1 Kernel Configuration" file.

CONFIG_HAVE_GENERIC_HARDIRQS=yCONFIG_CGROUPS=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_MEM_RES_CTLR=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set

For a live cd, it's running OK considering the little time I've used it. No real opinion pro or con as of yet. Guess the next kernel I compile on Gentoo ~Arch, I'll give this stuff a try.

Anyone have thoughts on the default Suse configs as opposed or related to what's being posted here by us Gentoo users?

EDIT: Upon further investigation, I couldn't find a /dev/cgroup directory, but I did find other similar cgroup sections in the kernel config file.

#
# IRQ subsystemCONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_MEM_RES_CTLR=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
#

#
# RCU Subsystem
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=yCONFIG_NET_CLS_CGROUP=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_MEM_RES_CTLR=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set

#
# Classification
#
CONFIG_NET_CLS_CGROUP=y

[Dont Panic]

If you're trying to see if the patch is enabled (or even there), look for the CONFIG_SCHED_AUTOGROUP setting.