Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
A [risky] way to make emerge really slow
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Mar 31, 2018 7:21 am    Post subject: A [risky] way to make emerge really slow Reply with quote

1) unmask and emerge cpulimit
2) limit to one process and one job at a time, for example:
Code:
MAKEOPTS="-j1" emerge --oneshot --jobs 1 cmake

3) After a while, identify running process name with "top" command:
Code:
top - 09:03:00 up  1:13,  3 users,  load average: 1.04, 0.59, 0.34
Tasks: 134 total,   2 running, 132 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  6.3 sy, 44.9 ni, 47.7 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem :  1025276 total,    27060 free,   134568 used,   863648 buff/cache
KiB Swap:  2097148 total,  2096884 free,      264 used.   789444 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6695 portage   39  19   38544  22548  11632 R  38.7  2.2   0:01.18 cc1plus
 6623 portage   39  19    5608   2748   2380 S   1.6  0.3   0:00.05 armv7a-hardfloa

4) start cpulimit in another window:
Code:
cpulimit -e cc1plus  -l 10
Process 6695 detected
Process 6695 dead!
Warning: no target process found. Waiting for it...
Process 6716 detected

During emerge the running cc1plus-process is slow and other useful jobs get more cpu :)


Last edited by Irre on Sat Mar 31, 2018 9:46 pm; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Sat Mar 31, 2018 10:55 am    Post subject: Reply with quote

Please tell us why (or in which cases) 'cpulimit' is better than 'nice'.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Mar 31, 2018 1:30 pm    Post subject: Reply with quote

mike155 wrote:
Please tell us why (or in which cases) 'cpulimit' is better than 'nice'.
I want to keep my PC cold or I run some applikation that is more important than emerge or ....
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Sat Mar 31, 2018 3:28 pm    Post subject: Reply with quote

Your system should be kept cool by its hardware cooling system.

Nice will handle giving other processes CPU preference over the Portage job.

According to the documentation, cpulimit supports being given the command to run, which is likely to be easier to use than resolving the pid manually. Also, based on a quick read of the source, it appears to be vulnerable to a buffer overflow when a process has a very large pid. It looks like cpulimit tries to track parental relations by walking proc, which is racy.

Some programs may react badly to unexpected signaling, particularly anything which uses ptrace as part of its tests. This would be a bug in the impacted program, but still a problem for the users harmed by the bad reaction.

Rather than play games with frequent signaling, why not use the cpu cgroup/scheduler integration to directly instruct the kernel not to run the processes so often?
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Mar 31, 2018 3:59 pm    Post subject: Reply with quote

To tell cpulimit to limit "emerge" is meaningless. You have to identify which subprocess is eating CPU, this is done as I described.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Sat Mar 31, 2018 4:59 pm    Post subject: Reply with quote

That supposes that only one type of command will be a CPU hog throughout the build. You would be better off limiting emerge and telling cpulimit to also limit all children.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Sat Mar 31, 2018 9:00 pm    Post subject: Reply with quote

Irre, thanks for introducing me to cpulimit. There are various use cases for such a program:
  1. To limit CPU power consumption in order to reduce CPU / system temperature or fan noise. This is a frequent application for me and I usually use cpupower to achieve that:
    Code:
    cpupower frequency-info -l    # show frequencies available
    cpupower frequency-set --max 2GHz

  2. To give more CPU time to other processes. This is also a frequent application for me and I usually use nice to achieve that:
    Code:
    nice emerge ...

    Sometimes I also use renice to fine-tune running applications.

  3. To limit power consumption of services (Apache, for example). I had such a use case 3 months ago and I used control groups and Systemd. It worked quite well. Systemd allows to limit control CPU usage, Memory Usage, Network bandwidth, etc. See this article for an introduction.

I'm not sure whether cpulimit can help me in these use cases:

Regarding use case 1: it's probably better to run the CPU with a constant (reduced) frequency, than to send start/stop signals to an application. Cpulimit might help on CPUs which do not support frequency scaling.

Regarding use case 2: I agree with Hu that it's better to let the OS scheduler control CPU usage.

And regarding use case 3: I would definitely recommend control groups and Systemd. I know that many people don't link Systemd, but Resource Management is something they did right.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Mar 31, 2018 9:07 pm    Post subject: Reply with quote

Hu wrote:
That supposes that only one type of command will be a CPU hog throughout the build. You would be better off limiting emerge and telling cpulimit to also limit all children.
Thank you! That is much better. I test with this command now:
Code:
MAKEOPTS="-j1" cpulimit -l 10 -m -- emerge --oneshot --jobs 1 cmake

Edit. Not so good! The system crashed!
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Sat Mar 31, 2018 9:32 pm    Post subject: Reply with quote

As a minor point, systemd provides a way to use the kernel's cgroups to limit resources in this area. You can use cgroups without systemd to achieve the same effect, though the interface for expressing your intent may not be as nice.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Mar 31, 2018 9:52 pm    Post subject: Reply with quote

Now I understand why cpulimit was masked. While testing, the system crashed hard and I had to run fsck on all file systems!

Don't try "cpulimit" if you have important systems running! :)
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Sun Apr 01, 2018 12:13 am    Post subject: Reply with quote

It's been over a decade since I've used a filesystem that needed fsck after a crash. Sounds like you don't have an X-Y problem so much as an X-$rest_of_alphabet problem...
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Sun Apr 01, 2018 12:28 am    Post subject: Reply with quote

In what way did the system crash? Does the system crash that same way if you skip cpulimit and let Portage go to full parallelism? If so, it may be as simple as that you have a problem with your system and cpulimit is not an adequate workaround for the underlying problem. (You mentioned earlier keeping the machine cold. Failure to manage temperature can cause crashes.)
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sun Apr 01, 2018 6:56 am    Post subject: Reply with quote

Unfortunately I have no logs as all logs have been written to tmpfs. I don't want to test on this failing machine any more as it is too important for me. It is used as a sambaserver and fileserver. Hardware is Cubieboard 20 (ARMv7). I have rebuilt world and I don't remember any previous crash.

Future tests will be on raspberry pi and gentoo under Virtualbox. :)

Edit: cpulimit was not masked on ~amd64. But useless. This command locked the system and all fans got crazy. I could enter "poweroff" command and system shutdown started but newer completed! I had to power off my WM!

do not run this command: MAKEOPTS="-j1" cpulimit -l 10 -m -- emerge --oneshot --jobs 1 cmake


Last edited by Irre on Sun Apr 01, 2018 8:08 am; edited 1 time in total
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2387
Location: Germania

PostPosted: Sun Apr 01, 2018 7:18 am    Post subject: Reply with quote

Ninja is a great resource hog escaping niceness. Using cgroups without systemd is quite easy.
See discussion about chromium build here (cgexec, cgset and cgcreate from dev-libs/libcgroup).
To properly build chromium not running oom on my build bot(4gb ram) I have these settings (I do not care if it is redundant :o):

Code:
ana ~ # grep ^CMAKE /etc/portage/make.conf
CMAKE_JOB_POOL_COMPILE=2
CMAKE_JOB_POOL_LINK=1

ana ~ # cat /etc/portage/env/chromium-j2.conf
MAKEOPTS="-j2 -l1" # -l1 allows build system to execute next step only if system load <= 1

_________________
hear hear
Back to top
View user's profile Send private message
fpemud
Guru
Guru


Joined: 15 Feb 2012
Posts: 349

PostPosted: Mon Apr 02, 2018 4:33 am    Post subject: Reply with quote

Quote:
See discussion about chromium build here (cgexec, cgset and cgcreate from dev-libs/libcgroup).


Can I use these methods in parallel with systemd?

Or

Must I use systemd to control all the cgroup related things if what I have is a systemd enabled system?
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Tue Apr 03, 2018 9:08 am    Post subject: Reply with quote

I emerged stable version of cpulimit and omitted -m option. Now the system didn't crash.

Test:
Code:
time MAKEOPTS="-j 9 --load-average=9" emerge -j 9 -v1 --quiet-build=y dev-libs/lzo dev-python/six
 dev-util/meson dev-python/extras dev-python/pyrsistent dev-python/lxml app-text/build-docbook-catalog
 dev-lang/perl x11-libs/pango dev-python/testtools dev-python/fixtures

real    5m41.133s
user    17m46.989s
sys     7m34.383s

time MAKEOPTS="-j 1 --load-average=1" emerge -j 1 -v1 --quiet-build=y dev-libs/lzo dev-python/six
 dev-util/meson dev-python/extras dev-python/pyrsistent dev-python/lxml app-text/build-docbook-catalog
 dev-lang/perl x11-libs/pango dev-python/testtools dev-python/fixtures

real    17m18.606s
user    11m47.087s
sys     5m19.202s
timing after I entered these commands
Code:
cpulimit -e cc1plus  -l 4&
cpulimit -e cc1  -l 4&
cpulimit -e perl  -l 4&

real    101m58.763s
user    11m43.422s
sys     5m18.940s


Conclusions

MAKEOPTS="-j 1 --load-average=1" and emerge -j 1 ...

increases emerge time from 6 to 17 minutes

Adding cpulimit commands increases time from 17 to 102 minutes in my test!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum