| View previous topic :: View next topic |
| Author |
Message |
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
 Posted: Tue Mar 14, 2017 2:00 pm Post subject: get a chromium build to play nice with SCHED_AUTOGROUP? |
 |
|
Like most gentoo users I compile a lot but it's usually not a problem because the compile is done in the background while I do other things. One annoying exception to this is www-client/chromium which bogs down my system a lot while it's building. Browsing the web is painfully slow and audio sometimes stutters. I used to assume this is just because chromium is a big project to build but other large projects like firefox doesn't cause the same slowdown. It's been a mystery for me for several years.
Today I looked into the problem and found the cause. My kernel is built with CONFIG_SCHED_AUTOGROUP. This is how the kernel option explains it:
| Quote: | This option optimizes the scheduler for common desktop workloads by
automatically creating and populating task groups. This separation
of workloads isolates aggressive CPU burners (like build jobs) from
desktop applications. Task group autogeneration is currently based
upon task session. |
from the "sched" manpage
http://man7.org/linux/man-pages/man7/sched.7.html
| Quote: | A new autogroup is created when a new session is created via
setsid(2); this happens, for example, when a new terminal window is
started. A new process created by fork(2) inherits its parent's
autogroup membership. Thus, all of the processes in a session are
members of the same autogroup.
Suppose that there are two autogroups competing for the same CPU
(i.e., presume either a single CPU system or the use of taskset(1) to
confine all the processes to the same CPU on an SMP system). The
first group contains ten CPU-bound processes from a kernel build
started with make -j10. The other contains a single CPU-bound
process: a video player. The effect of autogrouping is that the two
groups will each receive half of the CPU cycles. That is, the video
player will receive 50% of the CPU cycles, rather than just 9% of the
cycles, which would likely lead to degraded video playback.
|
This all sounds good and it's the reason I enabled the autogroup feature way back when it was first introduced. Checking /proc/[pid]/autogroup I can see that all make and gcc processes of a kernel make -j8 is in the same group. Unfortunately when I build chromium each build process gets it's own autogroup. Apparently chromium's custom "ninja" build system creates separate sessions for each build job. That negates the usefulness of autogroups.
But it gets worse. When autogroup is enabled the nice value of a process only apply within it's own autogroup. If there are two autogroups with a single process in each the nice value of those processes doesn't matter. They both get the same amount of cpu time. It's possible to set a nice value on an autogroup but this is something different from nice value on processes and tools like "nice" doesn't affect autogroup nice values.
I think that autogroup is a useful feature but the "auto" part doesn't always work so well and the changed behavior of nice with autogroups is an Achille's heel. Running "nice emerge chromium" with autogroups on give desktop responsiveness as bad as "emerge chromium" with autogroup off. I would prefer not to give up on autogroup because it works fine in most loads except chromium builds. Is there anything I can do to make it work better or should I just give up and fall back on manually setting priority on stuff? |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21595
|
| Posted: Wed Mar 15, 2017 1:50 am Post subject: |
|
|
That sounds like a bug in ninja. It should not be creating new autogroups for processes that are all part of the same overall job. Based on the comments in the ninja source, this is intentional, but looks very ill-considered to me: | Code: | // Put the child in its own session and process group. It will be
// detached from the current terminal and ctrl-c won't reach it.
// Since this process was just forked, it is not a process group leader
// and setsid() will succeed. |
The point of ctrl-c is that it will kill both the master (ninja) and the descendants (the build jobs), yet here Ninja is actively preventing that. It looks like you can prevent this by setting use_console_ to true (or by patching Ninja to remove the bogus setsid call). |
|
| Back to top |
|
|
geki
Advocate
Joined: 13 May 2004
Posts: 2387
Location: Germania
|
| Posted: Wed Mar 15, 2017 12:30 pm Post subject: |
|
|
I, too, use kernel's autogroup scheduler optimization and have issues similar to tholin during chromium's build process.
If anyone finds a solution(patch, build flag) for chromium build, please post it here, thanks a lot in advance!
edit #1
The cgroup solution by tholin below does the job; llvm and chromium built successfully without taking system down.
_________________
hear hear
Last edited by geki on Sat Mar 18, 2017 7:19 pm; edited 2 times in total |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Wed Mar 15, 2017 10:02 pm Post subject: |
|
|
You could try PORTAGE_IONICE_COMMAND="schedtool -B \${PID}", it should help even if you only have a non-MuQSS vanilla kernel.
(I've had llvm compiling in the background with this, and forgot it was running until the post-install file merge caused the game I was playing to slow down for a few seconds...) |
|
| Back to top |
|
|
geki
Advocate
Joined: 13 May 2004
Posts: 2387
Location: Germania
|
| Posted: Thu Mar 16, 2017 6:52 am Post subject: |
|
|
Ant P.
Let's see. I already reduced makejobs from -j4 to -j3 to succeed most of the times.
Now then, let's try that io nice with -j4 during system high load. 
That portage flag can be set in package.env, I guess.
_________________
hear hear |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Thu Mar 16, 2017 12:00 pm Post subject: |
|
|
After reading up on some old articles it seems that SCHED_AUTOGROUP was a somewhat controversial features when it was introduced. Some people argued that the heuristic was bad and that this kind of heuristic belongs in userspace instead. Perhaps they were right? The heuristic doesn't work so well but heuristics rarely do. That's something I can accept but what I can't accept is that the heuristic overrides manual user settings like nice.
| Ant P. wrote: | | You could try PORTAGE_IONICE_COMMAND="schedtool -B \${PID}", it should help even if you only have a non-MuQSS vanilla kernel. |
That don't work. Both nice value and non-realtime scheduling policies apply only within the autogroups. Since ninja creates a new autogroup for each build job the only thing that matters is the priority of the autogroup itself.
from man sched
| Quote: | The autogroup feature groups only processes scheduled under non-real-
time policies (SCHED_OTHER, SCHED_BATCH, and SCHED_IDLE). It does
not group processes scheduled under real-time and deadline policies.
Those processes are scheduled according to the rules described
earlier.
|
The possible solutions I've found so far are:
Just give up and disable autogroups. Then I can adjust priorities with tools like nice, schedtool and chrt. There are a bunch of tools like verynice, and (auto nice daemon) and Ananicy (ANother Auto NICe daemon) for auto applying a priority on a set of processes. That probably works but it seems a bit hacky to me.
https://github.com/poelzi/ulatencyd is a userspace daemon that seems similar to autogroups. It monitors the behavior of processes and auto sort them into cgroups. But I'm not interested in having more programs messing with my cgroups. If more than one program play around with cgroups the result is unpredictable and I already have problems keeping my current programs from stepping on each others toes. The author of ulatencyd also says that "the project is quite bitrotten these days". Probably because systemd keeps changing how it handles cgroups and this conflicts with every other user of cgroups.
Another possibility is to keep autogroup on but find a way to work around the problem with ninja. I'm not interested in patching ninja or using custom fixes for it. As the old saying goes "if you find a mushroom, look for more". There are probably other workloads doing the same thing as ninja. What I need is a way to override the autogroup.
From man sched
| Quote: | The use of the cgroups(7) CPU controller to place processes in
cgroups other than the root CPU cgroup overrides the effect of
autogrouping.
|
So what I need to do is put emerge in it's own cgroup. That way autogroup doesn't affect it. What I did was run "cgcreate -t <myuser>:<myuser> -a root:root -d 755 -g cpu:idle" to create a cpu group called idle. Then run "cgset -r cpu.shares=10 idle" to give the idle group 10 "shares" of the cpu. The root cpu group is set to 1024 so 10 is about 1% of total cpu. I tried to set it to 0 but for some reason the load of the idle group sometimes wouldn't use the cpu fully even when the cpu was idle. The real difference between 0% share and 1% share should be minimal anyway. I can then run emerge and any other cpu hog as "cgexec -g cpu:idle emerge chromium". According to some benchmarks I did this has even less impact on the rest of the system than running emerge with nice or "chrt --idle".
More info: http://blog.scoutapp.com/articles/2014/11/04/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups
The cgroup tools I use are part of dev-libs/libcgroup. It has a daemon to create the needed cpu cgroup on startup and another daemon for automatically assigning processes to cgroup kind of like those auto nice daemons.
Using a cgroup just for emerge and let everything else use autogroup is the best solution I've found so far. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Fri Mar 17, 2017 8:51 pm Post subject: |
|
|
| tholin wrote: | | That don't work. Both nice value and non-realtime scheduling policies apply only within the autogroups. Since ninja creates a new autogroup for each build job the only thing that matters is the priority of the autogroup itself. |
A userspace process can escape root-imposed nice(2) and sched_setscheduler(2) policies just by calling setsid? That seems like a security hole big enough to drive a ship through, do the kernel developers know about this? |
|
| Back to top |
|
|
geki
Advocate
Joined: 13 May 2004
Posts: 2387
Location: Germania
|
| Posted: Sun Sep 24, 2017 10:57 am Post subject: |
|
|
Next to using a cgroup, I found these useful environment flags from cmake documentation to set build jobs for compilation and linking.
I usually use 4 build jobs, but ninja seems to perform better with 2 build jobs (N - 2) wrt 4gb ram limitation.
And reduce linker build jobs even further for big libraries and/or gold linker.
| Code: | # cmake ninja specific
CMAKE_JOB_POOL_COMPILE=2
CMAKE_JOB_POOL_LINK=1 |
May it help you.
_________________
hear hear |
|
| Back to top |
|
|
|
Kernel & Hardware
