| View previous topic :: View next topic |
| Author |
Message |
halls
n00b
Joined: 22 Feb 2015
Posts: 4
|
 Posted: Sun Feb 22, 2015 7:39 pm Post subject: [SOLVED] Compiling RPi with DISTCC is working, but too slow |
 |
|
Hello all,
This is my first post on this forum, so please don't be too harsh on me 
Recently I've been installing Gentoo on my Raspberry Pi, following the official wiki pages (http://wiki.gentoo.org/wiki/Raspberry_Pi, http://wiki.gentoo.org/wiki/Raspberry_Pi_Quick_Install_Guide and http://wiki.gentoo.org/wiki/Raspberry_Pi_Cross_building). I have two i5 core PCs in my network, which are both running Gentoo and acting as DISTCC servers, and then the RPi as a client.
Everything seems to be working correctly, since I am monitoring distcc log files in the servers, and they both are getting jobs from RPi correctly, and packages are being correctly built on RPi (I've compiled almost everything that way). But recently I've noticed, looking at htop, that the workload on the servers is almost none. I mean, I think that maybe I could make DISTCC deliver more jobs to each server, so the CPU gets loaded with more jobs to process (and therefore the whole build process takes less time). In summary, it seems my DISTCC servers are not being used as much as they could be...
I believe I have not done anything special in the config, which I share here:
Configuration for PC#1 and PC#2:
/etc/conf.d/distccd
| Code: | DISTCCD_EXEC="/usr/bin/distccd"
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
DISTCCD_OPTS="--port 3632 --log-level notice --log-file /var/log/distcc/distccd.log -N 15 --allow 192.168.20.0/24 --allow 127.0.0.1" |
Configuration for Raspberry Pi client:
/etc/distcc/hosts
| Code: | nimbus ~ # distcc-config --get-hosts
# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
192.168.20.1/10,lzo 192.168.20.102/10,lzo |
/etc/portage/make.conf
| Code: | CFLAGS="-O2 -pipe -march=armv6j -mfpu=vfp -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"
CHOST="armv6j-hardfloat-linux-gnueabi"
# 2 quad core machines + 1 raspberry --> 2*(4+4+1) + 1 = 19
MAKEOPTS="-j19"
FEATURES="distcc"
USE="aalib alsa -cairo dbus directfb fbcon gif gtk gpm jpeg libcaca mp3 png sdl speex theora udev v4l xv X yuv4mpeg"
PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages" |
So I have been reading documentation on DISTCC, searched the forums and I don't see I might be missing anything special... I have been playing around with the "-j19" parameter, trying to set more jobs, less jobs... nothing is working. Compiling speeds up as for using DISTCC, but I believe it could be speeded even more, since I am not even using the servers to their 5% of CPU usage...
Any clues on this?
Thank you very much in advance!
Last edited by halls on Mon Feb 23, 2015 8:24 pm; edited 1 time in total |
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3137
|
| Posted: Sun Feb 22, 2015 8:12 pm Post subject: |
|
|
You invoke emerge from pi, right?
make sure your PI is NOT on distcc server list. Do not compile anything* on PI, it will choke pre- and post-processing of compilations done by the PC.
MAKEOPTS defines upper limit for compilation of a single package. Not many packages will have enough stuff inside to compile to use it all. Switch portage to parallel mode as well to fix it:
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=1" **
Also, distcc works better in pump mode: instead of calling it like "emerge blah blah blah" use "pump emerge blah blah blah"
Unfortunately it doesn't support hosts discovered with avahi (or didn't support when I was plaing with it), but it still bumps performance.
* some things will be compiled locally anyway for they disable distcc. Glibc seems to be one of those critical packages. And it will take time.
** distcc by default limits number of parallel compilations on any host. You might want to specify your own lmits. 2 per core will do a good start. This one is set on per-host basis in /etc/distcc/hosts file |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Sun Feb 22, 2015 8:30 pm Post subject: |
|
|
halls,
Run | Code: | | DISTCC_DIR=/var/tmp/portage/.distcc/ distccmon-text 5 |
on the Pi to see what distcc is doing.
I would be surprised if it could get 20 jobs in flight, even in pump mode. My Pi with 256Mb RAM manages 8 though.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
halls
n00b
Joined: 22 Feb 2015
Posts: 4
|
| Posted: Sun Feb 22, 2015 9:48 pm Post subject: |
|
|
Thank you for your replies!
@szatox:
| szatox wrote: | You invoke emerge from pi, right?
make sure your PI is NOT on distcc server list. Do not compile anything* on PI, it will choke pre- and post-processing of compilations done by the PC.
MAKEOPTS defines upper limit for compilation of a single package. Not many packages will have enough stuff inside to compile to use it all. Switch portage to parallel mode as well to fix it:
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=1" ** |
Yes, the emerge is done in the RPi, and no, the PI is not included in the hosts list (those two IPs listed correspond to my two PC machines, the PI ends in .191). I am not sure if I understood correctly what the EMERGE_DEFAULT_OPTS variable is for. I have been searching and found this web, where it says this:
| Code: | | To better utilize processors, one has to tell emerge to also run parallel jobs. This is done using the --jobs and --load-average options with emerge: |
Isn't it what you do with the MAKEOPTS="-j8" for example? I thought it was to tell portage to run 8 different make jobs in parallel. Or is it that what the --jobs does is to run different PORTAGE jobs? (not MAKE jobs)... Not sure if I am in the correct path...
Anyway, I am trying it right now, and I will let you know the results I get. Thanks for the advice 
| szatox wrote: | Also, distcc works better in pump mode: instead of calling it like "emerge blah blah blah" use "pump emerge blah blah blah"
Unfortunately it doesn't support hosts discovered with avahi (or didn't support when I was plaing with it), but it still bumps performance. |
Well regarding this, at first I was using it, but started to have many compilation fails. I read that if the systems you are compiling are not the same (say RPi vs PC) you might get errors due to path problems, etc. So finallly I just decided to remove it from the FEATURES list in the make.conf file.
@NeddySeagoon:
| NeddySeagoon wrote: | | I would be surprised if it could get 20 jobs in flight, even in pump mode. My Pi with 256Mb RAM manages 8 though |
Yes, in fact I was already monitoring the job dispatching from the RPi point of view with the distccmon-text tool and as you say, I can see up to 8 jobs in flight... but only from once in a while, that's why I was concerned.
Anyway, thanks both of you for your replies, I will try with the EMERGE_DEFAULT_OPTS and see what I can get! |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Sun Feb 22, 2015 10:01 pm Post subject: |
|
|
halls,
EMERGE_DEFAULT_OPTS just saves you typing.
--jobs= tells portage to run emerges in parallel if the dependency tree allows it.
Each --job can then have MAKEOPTS makes in progress concurrently.
Regarless of pump mode, the Pi still has to do the unpacking sending jobs to the helpers, the linking and installing.
So its busy even without running gcc.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
halls
n00b
Joined: 22 Feb 2015
Posts: 4
|
| Posted: Sun Feb 22, 2015 10:17 pm Post subject: |
|
|
Hi again,
I have been trying this, and see not much of a change...
| NeddySeagoon wrote: | halls,
EMERGE_DEFAULT_OPTS just saves you typing.
--jobs= tells portage to run emerges in parallel if the dependency tree allows it.
Each --job can then have MAKEOPTS makes in progress concurrently. |
Yes, that's exactly what I have seen now. Thanks for the explanation, now I get it.
| NeddySeagoon wrote: | Regarless of pump mode, the Pi still has to do the unpacking sending jobs to the helpers, the linking and installing.
So its busy even without running gcc. |
I see now. So I guess I won't have much of a CPU profit from the helpers, but anyway, even though the help is not much, is way more than having the PI do it all by itself.
Again, thanks for your replies! |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Sun Feb 22, 2015 10:30 pm Post subject: |
|
|
halls,
Some packages will cross compile correctly, so you can do the entire build on your helpers, then install the binary on the Pi.
There are a few wrinkles and some broken build systems that make it less than satisfactory.
e.g. Python builds some code, then tries to run it during the course of the build. Thats doomed to fail in a cross environment.
I've, tried all three approaches.
1) let the Pi do it all
2) cross compiling
3) cross distcc.
Cross distcc is the best in terms of my time.
For completeness, its also possible to set up a qemu arm environment on a PC.
qemu then emulates the ARM CPU. I'm told it works but its even slower than a real Pi.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
halls
n00b
Joined: 22 Feb 2015
Posts: 4
|
| Posted: Mon Feb 23, 2015 6:10 pm Post subject: |
|
|
Finally I have been compiling from yesterday, and everything seems to be fine, still not taking profit of all the computation capability, but enough for my needs.
Thank you for your help. This post can be marked as solved |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Mon Feb 23, 2015 6:33 pm Post subject: |
|
|
halls,
You can add solved to the title of the original post if you wish
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
