Compiling kernels with distcc

[Kream]

Compiling kernels with distcc

I test a number of kernels including vanillla, mandrake, lolo-sources, linux-beta and, of course, gentoo-sources so I find myself compiling the kernel lots of times. Even if you use distcc to emerge packages like I do, kernel compilations aren't distributed because the kernel make doesn't use /etc/make.conf. This is how I use distcc to radically speed up my compile times:
Overview

1. Set up distcc
   1. Emerge distcc on the computers
   2. Start distccd on the slave systems
2. Configure the kernel
3. Edit Makefile
4. make the kernel

Set up distcc
Set up distcc on all computers that you want to use, including the host computer. To accomplish this, a simple emerge command suffices:

Code: Select all

 emerge -u distcc

Once done, make sure you start the distccd daemon on all the computers that you want compilation to be distributed to. This is done by running this command on all of them:

Code: Select all

/etc/init.d/distccd start

Configure the kernel
Change to /usr/src/linux and configure the kernel using make menuconfig or make xconfig (does anyone still use make config ???? ). Save your config and and do a make dep.
Edit the Makefile
Fire up your favourite editor and load /usr/src/linux/Makefile. Find this section:

Code: Select all

#
# Include the make variables (CC, etc...)
#

And change "gcc" in the CC line to

Code: Select all

CC              = $(CROSS_COMPILE)distcc

Append this line to the section:

Code: Select all

DISTCC_HOSTS=slave1 slave2 slave3 ... slaveN

You can enter the hostnames of the computers if you're sure that they resolve. If they don't, you can use their IP addresses. It's best to enter the computers in descending order of CPU power availability.
If you want the local machine to participate in the compile, remember to start distccd on it as well and add localhost to the list of DISTCC_HOSTS. I don't recommend this, though, as the local machine will already be doing the preprocessing and co-ordinating.

Add DISTCC_HOSTS to the line which says

Code: Select all

export  VERSION PATCHLEVEL SUBLEVEL EXTRAVERSION KERNELRELEASE ARCH \

and save and exit.
Make the Kernel
When making the kernel, do a

Code: Select all

 make -jQ bzImage 

where Q is the number of parallel makes that you want to run. I recommend that Q be the number of slaves you have + 2. Examine the terminal's output to make sure that distcc is being called instead of gcc. If you're a gkrellm2 user like I am, you'll notice lots of data traffic outbound from your machine (preprocessed code) and a small amount of inbound traffic (compiled code).
I found my kernel being built *much* faster with distcc.

Aniruddha Shankar
Bangalore, India.

Vipassana Works.

[kyron]

To add to this wonderful document of yours, here is a script which I use each time I compile a kernel....just to make sue I don't forget anything

Code: Select all

make dep && make clean

make -j6 bzImage
make -j6 modules
make modules_install

mount /boot/
#keep a date stamped copy ;-P
cp /boot/bzImage /boot/bzImage.`date +%F_%k-%M`
cp arch/i386/boot/bzImage /boot/bzImage
umount /boot/

#Making sure alsa-driver 
export ALSA_CARDS=ens1370
emerge alsa-driver nvidia-kernel nvidia-glx
depmod -a
update-modules

[kyron]

One other thing I forgot, I also added ccache to the following line:

Code: Select all

 CC              = $(CROSS_COMPILE)ccache distcc

[matroskin]

does anyone still use make config ????

yes, i do. make config and make oldconfig are pretty good for configuring kernels throu scripts.

[kurifu]

For anyone who wishes to use distcc and genkernel to make their kernel, all you have to do is follow your normal distcc setup (emerge -v distcc, visit the distcc howto link provided at the end of the emerge), follow the instruction at the top of this thread to prepare your kernel make file and use the following command with genkernel:

Code: Select all

genkernel all --menuconfig --makeopts="-j5"

Obviously you can make a few changes to the genkernel command line, depending on how you wish to build the kernel. The important command is --makeopts="-j5" which actually sets up the number of parallel makes at once, needed to push some of the processes to your host.

[LordVan]

Thanks for the well written instructions I was looking for just this (my laptop is too slow to compile kernel in reasonable time -- and also tends to overheat )

[Yamakuzure]

And change "gcc" in the CC line to

Code: Select all

CC              = $(CROSS_COMPILE)distcc

Hi everybody!

If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with:

Code: Select all

# Make variables (CC, etc...)
CC              = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc

You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.

[Yord]

And change "gcc" in the CC line to

Code: Select all

CC              = $(CROSS_COMPILE)distcc

Hi everybody!

If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with:

Code: Select all

# Make variables (CC, etc...)
CC              = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc

You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.

Hi,

I came here for the RPI 3, but after much searching (albeit above didn't work for me, I think my distcc must be messed up somehow) I've found a simpler way!

e.g. make CC="distcc" bzImage

I don't know if you need to add the makeopts -- isn't that sourced through /etc/portage/make.conf?

[NeddySeagoon]

Yord,

Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.

For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with

Code: Select all

make modules install

though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not.

[Yord]

Yord,

Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.

For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with

Code: Select all

make modules install

though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not.

Ah, thanks. I actually have problems with compiling the way I did - distcc won't recognise the slave and it distributes locally instead.

Just to clarify, I'm trying to compile a kernel from the rpi, using distcc. There wouldn't be a problem with cluttering the build host in this way would there?

[NeddySeagoon]

Yord,

That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.

distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.

[Yord]

Yord,

That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.

distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.

Ah, Thanks!

I'm still having problems, my rpi conf looks like this:

Code: Select all

# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
192.168.1.90/8                                                                     

Code: Select all

# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"

/usr/lib/distcc/bin looks like this:

Code: Select all

lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc-4.9.3 -> /usr/bin/distcc
-rwxr-xr-x 1 root root 85 Aug 20 18:37 armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 c++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 cc -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 g++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 gcc -> armv7a-hardfloat-linux-gnueabi-wrapper

and on the PC doing the loading:

Code: Select all

# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
# 127.0.0.1                             

Code: Select all

# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"

and again, /usr/lib/distcc/bin looks like this (shouldn't matter?):

Code: Select all

lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc-4.9.3 -> /usr/bin/distcc
 

I've also rebuild the armv7a-hardfloat toolchain on my pc as a counter-measure.

So, for some reason distcc works perfectly when I'm using emerge, but it doesn't when I'm trying to compile a kernel (on the rpi). Is this the correct way to make a kernel?

Code: Select all

make ARCH=arm CC=distcc bcm2709_defconfig
make ARCH=arm CC=distcc  oldconfig
make ARCH=arm CC=distcc menuconfig
make ARCH=arm CC=distcc  -j8 bzImage modules
make ARCH=arm CC=distcc  -j8 modules_install

... and obviously copy the kernel img over.

EDIT: So, when I try compiling the kernel, I still get these errors:

Code: Select all

wonghau linux # make ARCH=arm CC=distcc -j8 bzImage modules
scripts/kconfig/conf  --silentoldconfig Kconfig
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
make[1]: 'include/generated/mach-types.h' is up to date.
  CHK     include/generated/timeconst.h
  CHK     include/generated/bounds.h
  CHK     include/generated/asm-offsets.h
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CC      arch/arm/kernel/smp.o
  CC [M]  drivers/bcma/main.o
distcc[6332] ERROR: compile arch/arm/kernel/smp.c on 192.168.1.90/8 failed
distcc[6332] (dcc_build_somewhere) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retrying locally
distcc[6332] Warning: failed to distribute arch/arm/kernel/smp.c to 192.168.1.90/8, running locally instead
  CC      block/cfq-iosched.o
distcc[6395] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/base/regmap/regmap-i2c.o
distcc[6401] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  mm/zsmalloc.o
distcc[6411] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/seqiv.o
distcc[6412] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  fs/9p/vfs_super.o
distcc[6414] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] ERROR: compile drivers/bcma/main.c on 192.168.1.90/8 failed
distcc[6368] (dcc_build_somewhere) Warning: remote compilation of 'drivers/bcma/main.c' failed, retrying locally
distcc[6368] Warning: failed to distribute drivers/bcma/main.c to 192.168.1.90/8, running locally instead
  CC [M]  drivers/base/regmap/regmap-spi.o
distcc[6453] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6332] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retried locally and got a different result.
  LD      arch/arm/kernel/built-in.o
  LD      kernel/trace/libftrace.o
  CC      kernel/trace/trace_events.o
  LD      drivers/block/built-in.o
distcc[6506] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/block/pktcdvd.o
distcc[6514] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/bcma/scan.o
distcc[6536] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'drivers/bcma/main.c' failed, retried locally and got a different result.
  CC [M]  drivers/bcma/core.o
distcc[6559] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC      kernel/trace/trace_export.o
distcc[6569] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  fs/9p/vfs_inode.o
distcc[6583] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/echainiv.o
distcc[6596] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/crypto_user.o
distcc[6608] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/cmac.o
distcc[6617] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

[laineg]

Code: Select all

CC              = $(CROSS_COMPILE)distcc

That won't work for the for the cross-compiling scenario, because distcc isn't CHOST-prefixed

Code: Select all

CC              = distcc $(CROSS_COMPILE)gcc

Tells distcc to execute gcc for the target, assuming the cross compiler is on your compile servers

And for ccache:

Code: Select all

CC              = ccache distcc $(CROSS_COMPILE)gcc

[ville.aakko]

Hi,

Thanks for this guide.

I've just noticed that for some reason distccd parallelizes really poorly (despite using -j5) - only one thread is compiling on the slave. Perhaps there's just too much overhead on the master to distribute compilation effectively / at all? (localhost is not in /etc/distcc/hosts)

I'm compiling a kernel for amd k6-3 box, and obviously it would be quite slow on the box itself.

Also, indeed the original instructions doesn't work if the slave computer is not the same arch. I needed to use:

Code: Select all

CC              = distcc $(CROSS_COMPILE)i586-pc-linux-gnu-gcc

It's a bit confusing, as one can not (or should not) set CROSS_COMPILE in the Makefile, as he/she is not cross-compiling in the Make process, i.e. on the target computer (amd k6-3 in my case) ?

Maybe this should be added to the distcc cross compilation wiki, as Kernels are quite a large compile. AFAIK it is otherwise outdated, too (the wrapper script is not needed anymore?).

EDIT: The wiki is missing instructions to use "--jobs" on the slave configuration. Now that I added it there, it's starting to give more jobs to the slave Still, quite some overhead and the master is not distributing processes as fast as it could - maybe 2-3 threads running on the slave (which could handle more, max jobs set to 6)

[NightMonkey]

Howdy. It seems that 'CC' has gone away. I see there's "HOSTCC' and 'HOSTCCX' in the Makefile. Anyone in the know on how best to enable Distcc kernel compilation these days? Thanks in advance.