Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage distcc spreading load unoptimally?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 91

PostPosted: Thu Sep 03, 2015 6:33 pm    Post subject: Portage distcc spreading load unoptimally? Reply with quote

So I have two computers, one is an ivybridge notebook, the other a more powerful haswell desktop. I decided to try my hand out at distcc which seems to slow things down, for now I'm just using it to compile stuff for my notebook using the desktop. I've tried various different ebuilds and they all amount to about a 140% increase in time taken to compile. distccmon-gui does reveal the desktop is getting jobs, but the load is pretty low compared to the notebook.

My make.conf on the weaker notebook:

Code:
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-march=ivybridge -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -maes -mpclmul -mpopcnt -mavx -msse4.2 -msse4.1 -mrdrnd -mf16c -mfsgsbase -mfxsr -mxsave -mxsaveopt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=6144 -mtune=ivybridge -fstack-protector-strong"
CFLAGS="${CFLAGS} -O2 -pipe"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
USE="bindist mmx sse sse2
   qt -kde -gnome -gtk -wxwidgets -xscreensaver -handbook

   -policykit -consolekit -acl -xdm-xauth -realtime -udisks -upower
   -bluetooth -nls -ncurses -cups -ppds -dvd -dvdr -cdda -cdr

   alsa pulseaudio
   nvidia
   portage

   truetype corefonts icu
   pcre
   
   abi_x86_64
   
   skypeapi
   
   lm_sensors
   
   -doc -hscolour

   "

PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages"

MAKEOPTS="-j25 -l8"
#MAKEOPTS="-j9"
FEATURES="distcc distcc-pump"

ACCEPT_KEYWORDS="~amd64"
ACCEPT_LICENSE="*"

EMERGE_DEFAULT_OPTS="--quiet-build=y"
CLEAN_DELAY=0

VIDEO_CARDS="intel nvidia"

INPUT_DEVICES="synaptics evdev"

PORTDIR_OVERLAY=/usr/local/portage

LINGUAS="en"

source /var/lib/layman/make.conf

GENTOO_MIRRORS="rsync://128.61.111.10/gentoo ftp://128.61.111.10/pub/gentoo rsync://ftp.snt.utwente.nl/gentoo http://91.121.189.201/gentoo-distfiles/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo"


output of distcc-config --get-hosts: "localhost/8 192.168.0.4/25"

Anything wrong with my settings that causes the desktop to get so little CPU load thus only slowing the overall thing down?
_________________
I will have you know that the male body has mechanisms to not get pregnant in cases of legitimate rape.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 34951
Location: 56N 3W

PostPosted: Thu Sep 03, 2015 6:50 pm    Post subject: Reply with quote

mi_unixbird,

Code:
output of distcc-config --get-hosts: "localhost/8 192.168.0.4/25"

That sends 8 jobs to localhost before any jobs are sent to the helper.

Put the entries in the other order to start with.
Also investigate pump mode, which allows parts of ./configure to be distributed too.

Code:
MAKEOPTS="-j25 -l8"
will be counter productive too. I think that's loadaverage <=8.
With 8 jobs on localhost, the load average will reach 8 and nothing will be distributed.

localhost still does the unpacking, linking and installing, regardless of the use of pump mode.
In some situations, like my Raspberry Pi with an AMD64 helper, I don't want any local compile jobs at all, so localhost is not in /etc/distcc/hosts.
Builds that fail to distribute will still fall back to localhost.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 91

PostPosted: Thu Sep 03, 2015 6:59 pm    Post subject: Reply with quote

NeddySeagoon wrote:
mi_unixbird,

Code:
output of distcc-config --get-hosts: "localhost/8 192.168.0.4/25"

That sends 8 jobs to localhost before any jobs are sent to the helper.

Put the entries in the other order to start with.
That sounds like a good idea to start with

Code:
MAKEOPTS="-j25 -l8"
will be counter productive too. I think that's loadaverage <=8.
With 8 jobs on localhost, the load average will reach 8 and nothing will be distributed.[/code]

Hmm, the wiki said you should "-jN -lM" where:

- N = the total number of logical cores * 2 + 1
- M = local number of logical cores

The weaker notebook interestingly enough has 8 logical cores while the stronger desktop only has 4. So 12*2+1=25.

Do you have a better number?
_________________
I will have you know that the male body has mechanisms to not get pregnant in cases of legitimate rape.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 34951
Location: 56N 3W

PostPosted: Thu Sep 03, 2015 7:18 pm    Post subject: Reply with quote

mi_unixbird,

Set -j(TotalLogicalCores+1) to start with.

Keep an eye on your compile farm and make sure nothing is driven into swapping.
If it is, reduce /LIMIT in /etc/distcc/hosts and reduce MAKEOPTS too.

Conversely increase things to get more parallel makes going.

A 256Mb Raspberry Pi can run MAKEOPTS=-j8 with a 6 core AMD64 accepting 8 jobs.
Its not a pretty sight doing gcc like that, which won't distribute.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 91

PostPosted: Thu Sep 03, 2015 7:57 pm    Post subject: Reply with quote

Yeah, I've seen to reliably been able to make it faster than a single one now that it's on -j13 instead of -j25. Wonder where that high number on the wiki game from, it seems counter-productive.
_________________
I will have you know that the male body has mechanisms to not get pregnant in cases of legitimate rape.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 34951
Location: 56N 3W

PostPosted: Thu Sep 03, 2015 8:25 pm    Post subject: Reply with quote

mi_unixbird,

The client needs to be able to do all the work that still needs to be done on the client wintout running out of RAM and CPU time.
Part of that work is sending compile jobs over the network and getting results back.

There is no point in setting MAKEOPTS above the lowest of the number of jobs the client can process or the number of jobs the helper(s) can run.
Hence you need some trial and error.
There is still a lot of things the helpers can't help with, so the optimum varies, depending on what is going on at the time.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum