Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compiling kernels with distcc
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Kream
n00b
n00b


Joined: 19 Apr 2002
Posts: 30
Location: New Delhi, India

PostPosted: Wed Dec 11, 2002 11:13 pm    Post subject: Compiling kernels with distcc Reply with quote

Compiling kernels with distcc

I test a number of kernels including vanillla, mandrake, lolo-sources, linux-beta and, of course, gentoo-sources so I find myself compiling the kernel lots of times. Even if you use distcc to emerge packages like I do, kernel compilations aren't distributed because the kernel make doesn't use /etc/make.conf. This is how I use distcc to radically speed up my compile times:
Overview

  1. Set up distcc

    1. Emerge distcc on the computers
    2. Start distccd on the slave systems

  2. Configure the kernel
  3. Edit Makefile
  4. make the kernel
Set up distcc
Set up distcc on all computers that you want to use, including the host computer. To accomplish this, a simple emerge command suffices:
Code:
 emerge -u distcc

Once done, make sure you start the distccd daemon on all the computers that you want compilation to be distributed to. This is done by running this command on all of them:
Code:
/etc/init.d/distccd start

Configure the kernel
Change to /usr/src/linux and configure the kernel using make menuconfig or make xconfig (does anyone still use make config ???? :roll: ). Save your config and and do a make dep.
Edit the Makefile
Fire up your favourite editor and load /usr/src/linux/Makefile. Find this section:
Code:
#
# Include the make variables (CC, etc...)
#

And change "gcc" in the CC line to
Code:

CC              = $(CROSS_COMPILE)distcc

Append this line to the section:
Code:

DISTCC_HOSTS=slave1 slave2 slave3 ... slaveN

You can enter the hostnames of the computers if you're sure that they resolve. If they don't, you can use their IP addresses. It's best to enter the computers in descending order of CPU power availability.
If you want the local machine to participate in the compile, remember to start distccd on it as well and add localhost to the list of DISTCC_HOSTS. I don't recommend this, though, as the local machine will already be doing the preprocessing and co-ordinating.

Add DISTCC_HOSTS to the line which says
Code:
export  VERSION PATCHLEVEL SUBLEVEL EXTRAVERSION KERNELRELEASE ARCH \
and save and exit.
Make the Kernel
When making the kernel, do a
Code:
 make -jQ bzImage
where Q is the number of parallel makes that you want to run. I recommend that Q be the number of slaves you have + 2. Examine the terminal's output to make sure that distcc is being called instead of gcc. If you're a gkrellm2 user like I am, you'll notice lots of data traffic outbound from your machine (preprocessed code) and a small amount of inbound traffic (compiled code).
I found my kernel being built *much* faster with distcc.

Aniruddha Shankar
Bangalore, India.

Vipassana Works.
Back to top
View user's profile Send private message
kyron
Apprentice
Apprentice


Joined: 26 Aug 2002
Posts: 198
Location: Montreal, Qc.

PostPosted: Fri Apr 11, 2003 1:45 am    Post subject: Reply with quote

To add to this wonderful document of yours, here is a script which I use each time I compile a kernel....just to make sue I don't forget anything :P
Code:
make dep && make clean

make -j6 bzImage
make -j6 modules
make modules_install

mount /boot/
#keep a date stamped copy ;-P
cp /boot/bzImage /boot/bzImage.`date +%F_%k-%M`
cp arch/i386/boot/bzImage /boot/bzImage
umount /boot/

#Making sure alsa-driver
export ALSA_CARDS=ens1370
emerge alsa-driver nvidia-kernel nvidia-glx
depmod -a
update-modules

_________________
M$ Windows: When in doubt, REBOOT
Linux: When in doubt, RTFM ;-)
Back to top
View user's profile Send private message
kyron
Apprentice
Apprentice


Joined: 26 Aug 2002
Posts: 198
Location: Montreal, Qc.

PostPosted: Fri Apr 11, 2003 1:49 am    Post subject: Reply with quote

One other thing I forgot, I also added ccache to the following line:
Code:
 CC              = $(CROSS_COMPILE)ccache distcc

_________________
M$ Windows: When in doubt, REBOOT
Linux: When in doubt, RTFM ;-)
Back to top
View user's profile Send private message
matroskin
Apprentice
Apprentice


Joined: 21 Jan 2003
Posts: 214

PostPosted: Wed Jul 16, 2003 1:31 pm    Post subject: Re: Compiling kernels with distcc Reply with quote

Kream wrote:
does anyone still use make config ???? :roll:
yes, i do. make config and make oldconfig are pretty good for configuring kernels throu scripts.
Back to top
View user's profile Send private message
kurifu
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jul 2003
Posts: 77

PostPosted: Sun Mar 28, 2004 10:00 pm    Post subject: distcc while using genkernel. Reply with quote

For anyone who wishes to use distcc and genkernel to make their kernel, all you have to do is follow your normal distcc setup (emerge -v distcc, visit the distcc howto link provided at the end of the emerge), follow the instruction at the top of this thread to prepare your kernel make file and use the following command with genkernel:

Code:
genkernel all --menuconfig --makeopts="-j5"


Obviously you can make a few changes to the genkernel command line, depending on how you wish to build the kernel. The important command is --makeopts="-j5" which actually sets up the number of parallel makes at once, needed to push some of the processes to your host.
Back to top
View user's profile Send private message
LordVan
Developer
Developer


Joined: 28 Nov 2002
Posts: 67
Location: Austria

PostPosted: Thu Nov 22, 2012 1:14 pm    Post subject: Reply with quote

Thanks for the well written instructions :) I was looking for just this (my laptop is too slow to compile kernel in reasonable time -- and also tends to overheat :( )
_________________
I don't suffer from insanity. I enjoy every minute of it.
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1898
Location: Bardowick, Germany

PostPosted: Tue Jul 12, 2016 7:32 am    Post subject: Re: Compiling kernels with distcc Reply with quote

Kream wrote:
And change "gcc" in the CC line to
Code:

CC              = $(CROSS_COMPILE)distcc
Hi everybody!

If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with:
Code:
# Make variables (CC, etc...)
CC              = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc
You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.
_________________
systemd - The biggest fallacies
Back to top
View user's profile Send private message
Yord
n00b
n00b


Joined: 21 Aug 2016
Posts: 3

PostPosted: Sun Aug 21, 2016 2:11 am    Post subject: Re: Compiling kernels with distcc Reply with quote

Yamakuzure wrote:
Kream wrote:
And change "gcc" in the CC line to
Code:

CC              = $(CROSS_COMPILE)distcc
Hi everybody!

If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with:
Code:
# Make variables (CC, etc...)
CC              = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc
You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.


Hi,

I came here for the RPI 3, but after much searching (albeit above didn't work for me, I think my distcc must be messed up somehow) I've found a simpler way! :)

e.g. make CC="distcc" bzImage

I don't know if you need to add the makeopts -- isn't that sourced through /etc/portage/make.conf?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 36153
Location: 56N 3W

PostPosted: Sun Aug 21, 2016 9:39 am    Post subject: Reply with quote

Yord,

Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.

For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with
Code:
make modules install
though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Yord
n00b
n00b


Joined: 21 Aug 2016
Posts: 3

PostPosted: Sun Aug 21, 2016 12:17 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Yord,

Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.

For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with
Code:
make modules install
though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not.


Ah, thanks. I actually have problems with compiling the way I did - distcc won't recognise the slave and it distributes locally instead.

Just to clarify, I'm trying to compile a kernel from the rpi, using distcc. There wouldn't be a problem with cluttering the build host in this way would there?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 36153
Location: 56N 3W

PostPosted: Sun Aug 21, 2016 12:52 pm    Post subject: Reply with quote

Yord,

That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.

distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Yord
n00b
n00b


Joined: 21 Aug 2016
Posts: 3

PostPosted: Mon Aug 22, 2016 8:39 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Yord,

That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.

distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.


Ah, Thanks! :D

I'm still having problems, my rpi conf looks like this:

Code:
# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
192.168.1.90/8                                                                     


Code:
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"


/usr/lib/distcc/bin looks like this:
Code:
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc-4.9.3 -> /usr/bin/distcc
-rwxr-xr-x 1 root root 85 Aug 20 18:37 armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 c++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 cc -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 g++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 gcc -> armv7a-hardfloat-linux-gnueabi-wrapper


and on the PC doing the loading:

Code:
# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
# 127.0.0.1                             


Code:
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"


and again, /usr/lib/distcc/bin looks like this (shouldn't matter?):
Code:
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc-4.9.3 -> /usr/bin/distcc
 


I've also rebuild the armv7a-hardfloat toolchain on my pc as a counter-measure.

So, for some reason distcc works perfectly when I'm using emerge, but it doesn't when I'm trying to compile a kernel (on the rpi). Is this the correct way to make a kernel?
Code:

make ARCH=arm CC=distcc bcm2709_defconfig
make ARCH=arm CC=distcc  oldconfig
make ARCH=arm CC=distcc menuconfig
make ARCH=arm CC=distcc  -j8 bzImage modules
make ARCH=arm CC=distcc  -j8 modules_install


... and obviously copy the kernel img over.

EDIT: So, when I try compiling the kernel, I still get these errors:
Code:

wonghau linux # make ARCH=arm CC=distcc -j8 bzImage modules
scripts/kconfig/conf  --silentoldconfig Kconfig
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
make[1]: 'include/generated/mach-types.h' is up to date.
  CHK     include/generated/timeconst.h
  CHK     include/generated/bounds.h
  CHK     include/generated/asm-offsets.h
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CC      arch/arm/kernel/smp.o
  CC [M]  drivers/bcma/main.o
distcc[6332] ERROR: compile arch/arm/kernel/smp.c on 192.168.1.90/8 failed
distcc[6332] (dcc_build_somewhere) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retrying locally
distcc[6332] Warning: failed to distribute arch/arm/kernel/smp.c to 192.168.1.90/8, running locally instead
  CC      block/cfq-iosched.o
distcc[6395] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/base/regmap/regmap-i2c.o
distcc[6401] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  mm/zsmalloc.o
distcc[6411] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/seqiv.o
distcc[6412] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  fs/9p/vfs_super.o
distcc[6414] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] ERROR: compile drivers/bcma/main.c on 192.168.1.90/8 failed
distcc[6368] (dcc_build_somewhere) Warning: remote compilation of 'drivers/bcma/main.c' failed, retrying locally
distcc[6368] Warning: failed to distribute drivers/bcma/main.c to 192.168.1.90/8, running locally instead
  CC [M]  drivers/base/regmap/regmap-spi.o
distcc[6453] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6332] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retried locally and got a different result.
  LD      arch/arm/kernel/built-in.o
  LD      kernel/trace/libftrace.o
  CC      kernel/trace/trace_events.o
  LD      drivers/block/built-in.o
distcc[6506] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/block/pktcdvd.o
distcc[6514] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  drivers/bcma/scan.o
distcc[6536] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'drivers/bcma/main.c' failed, retried locally and got a different result.
  CC [M]  drivers/bcma/core.o
distcc[6559] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC      kernel/trace/trace_export.o
distcc[6569] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  fs/9p/vfs_inode.o
distcc[6583] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/echainiv.o
distcc[6596] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/crypto_user.o
distcc[6608] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CC [M]  crypto/cmac.o
distcc[6617] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum