Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
distcc cross-compilation is not very helpful.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM
View previous topic :: View next topic  
Author Message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Thu Jul 11, 2019 11:02 am    Post subject: distcc cross-compilation is not very helpful. Reply with quote

Many source files are compiled locally on qemu aarch64 chroot instead of gentoo amd64.

Code:
> env DISTCC_DIR='/mnt/data/Backup/rpi3b-plus/var/tmp/portage/.distcc' distccmon-text 1 | grep -v -e '^$'
 18353  Compile     conftest.cpp                                  localhost[0]
 18415  Compile     conftest.cpp                                  localhost[0]
 18695  Compile     conftest.c                                    localhost[0]
 18719  Compile     conftest.c                                    localhost[0]
 18778  Compile     conftest.cpp                                  localhost[0]
 18778  Compile     conftest.cpp                                  localhost[0]
 18869  Compile     conftest.cpp                                  localhost[0]
 18909  Compile     conftest.cpp                                  localhost[0]
 18965  Compile     conftest.cpp                                  localhost[0]
 19055  Compile     conftest.cpp                                  localhost[0]
 20618  Compile     version.c                                     localhost[0]
 21135  Compile                                                   localhost[0]
 21139  Compile                                                   localhost[1]
 21308  Compile     gengtype-lex.c                                localhost[0]
 21310  Compile     gengtype-parse.c                              localhost[1]
 21328  Compile     genhooks.c                                    localhost[2]
 21300  Compile     errors.c                                      localhost[3]
 21308  Compile     gengtype-lex.c                                localhost[0]
 21310  Compile     gengtype-parse.c                              localhost[1]
 21328  Compile     genhooks.c                                    localhost[2]
 21300  Compile     errors.c                                      localhost[3]
 21308  Compile     gengtype-lex.c                                localhost[0]
 21310  Compile     gengtype-parse.c                              localhost[1]
 21328  Compile     genhooks.c                                    localhost[2]
 21300  Compile     errors.c                                      localhost[3]
 21308  Compile     gengtype-lex.c                                localhost[0]
 21328  Compile     genhooks.c                                    localhost[2]
 21298  Compile     genmodes.c                                    localhost[3]
 21308  Compile     gengtype-lex.c                                localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21781  Compile                                                   localhost[2]
 21298  Compile     genmodes.c                                    localhost[3]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21320  Compile     gengenrtl.c                                   localhost[2]
 21298  Compile     genmodes.c                                    localhost[3]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21320  Compile     gengenrtl.c                                   localhost[2]
 21298  Compile     genmodes.c                                    localhost[3]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21320  Compile     gengenrtl.c                                   localhost[2]
 21298  Compile     genmodes.c                                    localhost[3]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21990  Compile                                                   localhost[2]
 21994  Compile                                                   localhost[3]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21352  Compile     genchecksum.c                                 localhost[2]
 21318  Compile     gengtype-state.c                              localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 21352  Compile     genchecksum.c                                 localhost[2]
 22154  Compile     min-insn-modes.c                              localhost[3]
 22149  Compile     genmddeps.c                                   localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 22152  Compile     genconstants.c                                localhost[2]
 22154  Compile     min-insn-modes.c                              localhost[3]
 22149  Compile     genmddeps.c                                   localhost[0]
 21303  Compile     gengtype.c                                    localhost[1]
 22152  Compile     genconstants.c                                localhost[2]
 22154  Compile     min-insn-modes.c                              localhost[3]
 22149  Compile     genmddeps.c                                   localhost[0]
 22232  Compile                                                   localhost[1]
 22152  Compile     genconstants.c                                localhost[2]
 22158  Compile     inchash.c                                     localhost[3]
 22149  Compile     genmddeps.c                                   localhost[0]
 22161  Compile     genenums.c                                    localhost[1]
 22152  Compile     genconstants.c                                localhost[2]
 22158  Compile     inchash.c                                     localhost[3]
 22150  Compile     read-md.c                                     localhost[0]
 22161  Compile     genenums.c                                    localhost[1]
 22200  Compile                                                   localhost[2]
 22158  Compile     inchash.c                                     localhost[3]
 22150  Compile     read-md.c                                     localhost[0]
 22161  Compile     genenums.c                                    localhost[1]
 22158  Compile     inchash.c                                     localhost[3]
 22150  Compile     read-md.c                                     localhost[0]
 22161  Compile     genenums.c                                    localhost[1]
...


If I filter out local compilation, I get the following result.

Code:
> env DISTCC_DIR='/mnt/data/Backup/rpi3b-plus/var/tmp/portage/.distcc' distccmon-text 1 | grep -v -e '^$' -e 'localhost\['
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26629  Compile     aarch64-c.c                               192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
 26605  Compile     c-spellcheck.cc                           192.168.80.10[0]
...


As you can see in the following output, gcc is somewhat friendly with distcc, but many packages are not.

Code:
> env DISTCC_DIR='/mnt/data/Backup/rpi3b-plus/var/tmp/portage/.distcc' distccmon-text 1 | grep -v -e '^$'
 30198  Compile     combine.c                                 192.168.80.10[0]
 30203  Compile     combine-stack-adj.c                       192.168.80.10[1]
 26605  Compile     c-spellcheck.cc                               localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 28707  Compile     builtins.c                                    localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 30203  Compile     combine-stack-adj.c                       192.168.80.10[1]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 28707  Compile     builtins.c                                    localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 30203  Compile     combine-stack-adj.c                           localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 30203  Compile     combine-stack-adj.c                           localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 30203  Compile     combine-stack-adj.c                           localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 30203  Compile     combine-stack-adj.c                           localhost[3]
 30198  Compile     combine.c                                 192.168.80.10[0]
 29769  Compile     symtab.c                                      localhost[0]
 29989  Compile     cgraphclones.c                                localhost[1]
 29955  Compile     cgraphunit.c                                  localhost[2]
 30203  Compile     combine-stack-adj.c                           localhost[3]
...


How can I make emerge compile more source files remotely? Would Configuring distcc to cross-compile correctly help?
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 445

PostPosted: Thu Jul 11, 2019 2:29 pm    Post subject: Reply with quote

You could make a qemu based setup where you precompile and use binary distribution instead.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 55289
Location: 56N 3W

PostPosted: Thu Jul 11, 2019 6:39 pm    Post subject: Reply with quote

crocket,

Post the /etc/distcc/hosts file form the Pi, the
Code:
emerge --info
from the Pi and the /etc/conf.d/distccd from the helpers.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Thu Jul 11, 2019 10:43 pm    Post subject: Reply with quote

NeddySeagoon wrote:
crocket,

Post the /etc/distcc/hosts file form the Pi, the
Code:
emerge --info
from the Pi and the /etc/conf.d/distccd from the helpers.


/etc/distcc/hosts from Pi

Code:
192.168.80.10,cpp,lzo


emerge --info on qemu aarch64 chroot because emerge doesn't work properly on Pi.

Code:
Portage 2.3.62 (python 3.6.5-final-0, default/linux/arm64/17.0, gcc-8.2.0, glibc-2.29-r2, 4.19.52-gentoo aarch64)
=================================================================
System uname: Linux-4.19.52-gentoo-aarch64-with-gentoo-2.6
KiB Mem:    32841832 total,  30837996 free
KiB Swap:    9109496 total,   9109496 free
Timestamp of repository gentoo: Wed, 10 Jul 2019 23:30:01 +0000
Head commit of repository gentoo: 80ed751db3e893e3df598e3eeb18a645749c91da
Timestamp of repository haskell: Sat, 06 Jul 2019 20:46:01 +0000
Head commit of repository haskell: efe4c420cc01a65878b481f76da147615ca5f81e

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.31.1 p7) 2.31.1
distcc 3.3.2 aarch64-unknown-linux-gnu [enabled]
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.28.2-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.38.3-r1::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.31.1-r6::gentoo
sys-devel/gcc:            8.2.0-r6::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts:
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

crocket-overlay
    location: /mnt/data/git/gentoo-overlay/crocket-overlay
    masters: gentoo haskell

haskell
    location: /var/db/repos/haskell
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/haskell.git
    masters: gentoo

ACCEPT_KEYWORDS="arm64"
ACCEPT_LICENSE="@FREE"
CBUILD="aarch64-unknown-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="aarch64-unknown-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distcc distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j17 -l8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl alsa arm64 berkdb bzip2 cli crypt cxx dri flac fortran gdbm iconv ipv6 jack jpeg2k libtirpc mms mp3 ncurses nls nptl openmp opus pam pcre readline seccomp ssl tcpd unicode vorbis xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="edsp neon thumb thumb2 v4 v5 v6 v7 v8 vfp vfp-d32 vfpv3 vfpv4" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="fbdev dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


/etc/conf.d/distccd on 192.168.80.10

Code:
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical --log-file /var/log/distccd.log"

# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.80.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.80.10"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 55289
Location: 56N 3W

PostPosted: Fri Jul 12, 2019 7:21 pm    Post subject: Reply with quote

crocket,

/etc/distcc/hosts from Pi:
192.168.80.10,cpp,lzo

allows the Pi to send 4 concurrent jobs to 192.168.80.10, using pump mode if you like.
Any further jobs will run on localhost, even though its not listed there, e.g. if MAKEOPTS allows more that 4 jobs on the Pi.

crocket wrote:
emerge --info on qemu aarch64 chroot because emerge doesn't work properly on Pi.

Why would you expect a broken emerge to use distcc properly?
That needs to be fixed first.

crocket wrote:
/etc/conf.d/distccd on 192.168.80.10
is missing the --jobs option.
That's not wrong, the default is threads+2. If you try to send 192.168.80.10 more than threads+2 jobs they will be rejected.

We don't know MAKEOPTS on the Pi but distcc will only send 4 jobs to 192.168.80.10
We don't know threads+2 on 192.168.80.10 but if there are too many jobs floating around, localhost will be used.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Fri Jul 12, 2019 9:36 pm    Post subject: Reply with quote

NeddySeagoon wrote:
crocket,

/etc/distcc/hosts from Pi:
192.168.80.10,cpp,lzo

allows the Pi to send 4 concurrent jobs to 192.168.80.10, using pump mode if you like.
Any further jobs will run on localhost, even though its not listed there, e.g. if MAKEOPTS allows more that 4 jobs on the Pi.

crocket wrote:
emerge --info on qemu aarch64 chroot because emerge doesn't work properly on Pi.

Why would you expect a broken emerge to use distcc properly?
That needs to be fixed first.

crocket wrote:
/etc/conf.d/distccd on 192.168.80.10
is missing the --jobs option.
That's not wrong, the default is threads+2. If you try to send 192.168.80.10 more than threads+2 jobs they will be rejected.

We don't know MAKEOPTS on the Pi but distcc will only send 4 jobs to 192.168.80.10
We don't know threads+2 on 192.168.80.10 but if there are too many jobs floating around, localhost will be used.


/etc/distcc/hosts on qemu aarch64 chroot

Code:
192.168.80.10/8,cpp,lzo


I broke emerge on Pi on purpose by removing /usr/portage on Pi. I don't want to wear microSD card by keeping /usr/portage on it. In qemu aarch64 chroot, gentoo desktop's /usr/portage is mounted on /usr/portage.
After everything is compiled, I push the contents of qemu aarch64 chroot to Pi via "zfs send" and ssh.

Code:
MAKEOPTS="-j17 -l8"


on qemu aarch64 chroot and Pi because AMD FX-8300 has 8 threads. 192.168.80.10 is localhost. I haven't applied Configuring distcc to cross-compile correctly, yet.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7472

PostPosted: Sat Jul 13, 2019 5:29 pm    Post subject: Reply with quote

crocket wrote:
Code:
MAKEOPTS="-j17 -l8"

Either you are showing the amd MAKEOPTS, and it's totally useless, because this is use when the amd will emerge something, when the Pi emerge something, the MAKEOPTS of the Pi will be used
Or you are showing the Pi settings, but with -j17, you're insane :)
ps: i'm kidding, even for the amd that -j17 is insane :D
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 55289
Location: 56N 3W

PostPosted: Sat Jul 13, 2019 8:25 pm    Post subject: Reply with quote

crocket,

The -l8 in MAKEOPTS is not always honoured.

That leaves -j17 being over optimistic for an 8 thread system.
On a large C++ package, that might try to use 2G RAM per C++ thread. You get to keep all the pieces then.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Sat Jul 13, 2019 10:43 pm    Post subject: Reply with quote

https://wiki.gentoo.org/wiki/Distcc says

Quote:
Set the MAKEOPTS variable and FEATURES variable as shown below.

A common strategy is to

  • set the value of N to twice the number of total (local + remote) CPU cores + 1, and
  • set the value of M to the number of local CPU cores

The use of -lM in the MAKEOPTS variable will prevent spawning too many tasks when some of the distcc cluster hosts are unavailable (increasing the amount of simultaneous jobs on the other systems) or when an ebuild is configured to disallow remote builds (such as with gcc). This is accomplished by refusing to start additional jobs when the system load is at or above the value of M.
Code:
# Replace N and M with the right value as calculated previously
MAKEOPTS="-jN -lM"
FEATURES="distcc"

For instance, when there are two quad-core host PCs running distccd and the local PC has a dual core CPU, then the MAKEOPTS variable could look like this:
Code:
# 2 remote hosts with 4 cores each = 8 cores remote
# 1 local host with 2 cores = 2 cores local
# total number of cores is 10, so N = 2*10+1 and M=2
MAKEOPTS="-j21 -l2"


So, for an 8-thread CPU, the value of N is ( 2 x 8 ) + 1 = 17, and the value of M is 8 because qemu aarch64 chroot distributes its workload to amd64 gentoo system on localhost.
What do you recommend as the value of MAKEOPTS? -j 8?

The real problem is that about 90~99% of source files are compiled on qemu aarch64 chroot instead of amd64 gentoo system.

Things I fixed after submitting this post

  • /etc/distcc/hosts
    Code:
    192.168.80.10,cpp,lzo

    was changed to
    Code:
    192.168.80.10/8,cpp,lzo

  • Cross-compilation was broken because I didn't choose aarch64-unknown-linux-gnu-8.3.0 via "eselect gcc set" after removing aarch64-unknown-linux-gnu-8.2.0. So, I chose aarch64-unknown-linux-gnu-8.3.0.

Things I will change

I don't yet know whether fixing those will be sufficient.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 55289
Location: 56N 3W

PostPosted: Sun Jul 14, 2019 9:43 am    Post subject: Reply with quote

crocket,

The
Code:
192.168.80.10/8,cpp,lzo
will help a lot.

Configuring distcc to cross-compile correctly will help with some cross compile packages that have broken build systems.
Its not just discc that needs to be fixed, local cross compiling can be an issue too, where the hosts toolchain gets called.

Its a bit hackish but you can add CC= and so on to make.conf to call the right toolchain.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on ARM All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum