Compilation on low end hardware w. distcc, pump, ccache, lxd

[Tablestone]

##
## Compilation on low end computers with distcc, pump, ccache and lxd/lxc containers.
##

1. DESCRIPTION

This is a simple benchmarking study of different compilation configurations with distcc, pump, ccache and lxd/lxc containers.

2. BACKGROUND

I am an experienced Linux user, but a beginner with the Gentoo distro. While running emerge and reading the forums, it became apperent that compiling huge packages, like gcc or a DE, is a very time consuming endeavour. This other day gcc emerged for upgrade. Tried to compile it but the computer shutdown because of too high temprature - Yah! (Later I found out about the 'ondemand' processor configuration and added it as default mode.) So I surveyed my options and compiled my results in this little write-up.

There are no detailed instructions how to install the tools of this setup as they are available elsewhere.

Please feel free to comment, correct or give any suggestions/tweaks.

3. HARDWARE LINE-UP

Computers: 1 master + 2 slaves

Master OS: Gentoo
Processor: Intel Core2 Duo 1.8GHz
Memory: 2G
Swap: 4G

Slave OS: Ubuntu 16.04 + LXD/LXC
Processor: Intel Core2 Duo 2.0GHz & 2.13GHz
Memory: 4G
Swap: 8G

All computers are connected with wire to the same network switch.

4. SOFTWARE INSTALLED

Gentoo Master: distcc, ccache

Ubuntu Slaves: LXD (apt-get install lxd zfs) and latest gentoo (amd64) containers (lxc images list images:)

Containers: distcc ccache

5. NETWORK CONFIG

Ubuntu Slaves:

lxd init: container reachable from the internet=yes

Gentoo Master:

ip route add [container subnet 1]/24 via [ubuntu slave 1 ip]
ip route add [container subnet 2]/24 via [ubuntu slave 2 ip]

6. DISTCC CONFIG

6.1 MAIN CONFIGURATION

Gentoo Master and Slave containers:

DISTCCD_OPTS=""
DISTCCD_EXEC="/usr/bin/distccd"
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level notice --log-file /var/log/distccd.log"
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24 --allow 127.0.0.1"
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"

-- Fix logfile

touch /var/log/distccd.log
chown distcc:adm /var/log/distccd.log

-- Start distccd

rc-update add distccd default
rc-service distccd start

Comment: If you want localhost to run distcc you have to set "--allow 127.0.0.1" and add 127.0.0.1 in your DISTCC_HOSTS environment variable. Just setting "localhost" in DISTCC_HOSTS will not trigger distcc on localhost.

For debugging distcc set: export DISTCC_VERBOSE=1

6.2 RUNNING THE TEST

You can use the command distcc-config but the best option is setting the DISTCC_HOSTS environment variable.

Gentoo Master:

export DISTCC_HOSTS="--randomize localhost/2 X.X.X.A Y.Y.Y.A ..."
export DISTCC_HOSTS="--randomize localhost/2 X.X.X.A,lzo,cpp Y.Y.Y.A,lzo,cpp ..." (for running pump)

Slave containers: -

7. ABOUT THE TEST

The 'time' command is prefixed to all compile commands. I have not tried to tweak any special variables, like ccache cache size, except the N-value in 'make -jN'. For low end computers, distcc + lxc containers seems to be the best option. I have tried compiling the Gentoo kernel with distcc running on Ubuntu 16.04, but got compiling errors - even with the same gcc version. Installing and configuring a Gentoo toolchain on Ubuntu was daunting and would probably mess up my Ubuntu installation. That's why I took the fast and easy way and installed LXD on Ubuntu, downloaded a Gentoo container image. Voila - the toolchain was now identical, and the kernel could compile without errors.

The gentoo containers are running on the 2 Ubuntu "slaves". The 'lxc' value in the table headers below is the amount of containers utilized for each compile round.

The kernel configuration is customized and slimmed for the computer, if you think the compile time is oddly low.

Testing with distcc,pump and ccache together gives alot of warnings like: "... cannot use distcc_pump on already preprocessed file (such as emitted by ccache)".

As you will see in my results, in this small setup of computers, pump or ccache is not helping getting the numbers down.

8. REMEMBER

After each compilation with ccache you have to run 'ccache -c' and 'ccache -C' to empty the cache before next run. And of course run: make clean.

9. EMERGE

If you are going to use distcc with emerge you have to configure your make.conf:

Example configuration:

FEATURES="distcc"
MAKEOPTS="-jN -lM" (N=your calculated number, M=cores on master computer)
CFLAGS="... -march=YOUR_PROCESSOR_TYPE", example: -march=core2, NOT: -march=native (!)

10. CALCULATING N (in make -jN)

Default formula for N is: computers * cores * 2 + 1

But when using LXC containers, that formula will not suit that well. So I had to find value that gives the lowest compile time. It's about hammering the distcc-slaves and keeping them busy all the time, but not to much or you will start to DDos yourself. For this setup, N=45 seems to be the peak value for master.

If you get to many "IO Errors" or "failed to distribute" during make, you probably have to lower the N.

11. FUTURE STUDIES

- Could I use my friends computers over the Internet to lower the compile time even further?

- distcc seems to be creating new network connections for every job. If persistent connections could be set up, could that speed things up more?

12. THE BENCHMARK RESULTS

[eccerr0r]

I've found:

- there's a lot of "junk" that can't be distcc'ed - so it buys nothing
- Network connection speed hurts - anything less than 100Mbit can really hurt.

My distcc system that I REALLY noticed this behavior:

P3-933 Fujitsu Stylistic 512MB - distcc master. Experimenting with ethernet and wifi.
Core2 Quad 9550S 8GB - Distcc slave (one 32 bit VM, and 64-bit crossdev)
Core i7-2700k 8GB - Distcc slave (64-bit crossdev)

Builds are still very slow on the P3, transferring distcc files to the slaves took a really long time on wifi, and it almost seemed like it didn't really help overall. Behavior is very similar on my Atom 1.6GHz.

Core2 machines that you have are considerably faster than the p3...

I ended up doing Jaglover's suggestion to NFS mount the slow machine and build on the fast one... This might not be an option for you as your machines are on the same order of magnitude, but for me the p3 and many of my other machines aren't even worth being in my distcc pool because of the speed ratios...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?