Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Need help getting distcc working (solved)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sat Nov 03, 2012 11:37 pm    Post subject: Need help getting distcc working (solved) Reply with quote

I'm trying to get distcc working with a strict client/server environment.

Here is the configuration I have so far:

Server Configuration

IP Address:
Code:
192.168.10.5


/etc/conf.d/distccd:
Code:
# /etc/conf.d/distccd: config file for /etc/init.d/distccd
                               
DISTCCD_OPTS=""                 

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"
                               
# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"

# SECURITY NOTICE:
# It is HIGHLY recomended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.10.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.10.5"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"


/etc/make.conf:
Code:
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-march=core2 -O2 -pipe"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
USE="X a52 aac alsa apng bidi bsf cdda cdio dbus dts dvd fat ffmpeg flac gif
     gimp gnome gtk gudev hwdb jpeg libkms live lua mad matroska minizip mjpeg
     mmx mp3 mp4 mpeg mplayer nvidia ogg opengl png quicktime sse sse2 stream
     theora tiff truetype udev vcd vlm vorbis wav wma x264 xorg xv xvid"
MAKEOPTS="-j3"
INPUT_DEVICES="evdev"
VIDEO_CARDS="nvidia"
PORTDIR_OVERLAY="/usr/local/portage"
FEATURES="distcc"


/etc/distcc/hosts:
Code:
192.168.10.5


/etc/init.d/distccd is running

Notes:
This machine already has the toolchain build for the architecture I'm building for. I've been using the distcc monitor and haven't seen any activity.

Client Configuration

IP Address:
Code:
192.168.10.205


/etc/conf.d/distccd:
Code:
# /etc/conf.d/distccd: config file for /etc/init.d/distccd

DISTCCD_OPTS=""

# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"

# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"

# set this option to run distccd with extra parameters
# Default port is 3632.  For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"

# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL  [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical
                               
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
                               
# SECURITY NOTICE:             
# It is HIGHLY recomended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example:  --allow 192.168.0.0/24
# example:  --allow 192.168.0.5 --allow 192.168.0.150
# example:  --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"

# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"


/etc/make.conf:
Code:
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="armv6j-hardfloat-linux-gnueabi"

MAKEOPTS="-j5"

FEATURES="parallel-fetch buildpkg distcc"

#GENTOO_MIRRORS="http://intelminer.com/gentoo"

#SYNC="rsync://intelminer.com/gentoo-portage"

ACCEPT_LICENSE="*"

EMERGE_DEFAULT_OPTS="--quiet-build=n"

PORTAGE_NICENESS="15"

PKGDIR="/var/binary"

USE="gif jpeg mad mp3 png quicktime theora x264-static xv xvid"


/etc/distcc/hosts:
Code:
192.168.10.5


/etc/init.d/distccd is running

Notes:
This is a raspberry pi. When I run emerge, it does compile whatever I ask it to, but it is obvious that it's doing it itself and not using distcc.

What am I missing here?


Last edited by Bigun on Sun Nov 04, 2012 8:06 pm; edited 1 time in total
Back to top
View user's profile Send private message
Jaglover
Advocate
Advocate


Joined: 29 May 2005
Posts: 4805
Location: Saint Amant, Acadiana

PostPosted: Sat Nov 03, 2012 11:46 pm    Post subject: Reply with quote

You do need distcc running on server,
Code:
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.10.0/24"
is sufficient.
You do not need distcc running on client, but you may want to run emerge in pump mode.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sat Nov 03, 2012 11:57 pm    Post subject: Reply with quote

Pump mode? I looked into that briefly and I think that may be an alternative to distcc, correct?
Back to top
View user's profile Send private message
Jaglover
Advocate
Advocate


Joined: 29 May 2005
Posts: 4805
Location: Saint Amant, Acadiana

PostPosted: Sun Nov 04, 2012 12:09 am    Post subject: Reply with quote

Nope, pump is an addition to distcc. Just run "pump emerge" instead of plain emerge. If your crosscompiler is up to the task you'll see most of hard work done by server.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32443
Location: 56N 3W

PostPosted: Sun Nov 04, 2012 12:14 am    Post subject: Reply with quote

Bigun,

Pump mode allows more than compiles to be offloaded.

On my Pi /etc/distcc/hosts contains
Code:
 192.168.100.20/8,lzo,cpp
which tells it to send a maximum of 8 jobs to my Phenom II 1090.
lzo is for compression, which pump mode requires and cpp allows the use of pump mode.

make.conf on the Pi contains
Code:
# -j8 for distcc, -j1 or -j2 when there are no helpers
MAKEOPTS="-j2"


The helper is set up to listem on 192.168.0.0/16 because its on several subnets

Code:
pump emerge kde-meta
on the Pi sends most of the work to the cross compiler on the AMD box.
No - you can't run KDE on a Pi but you get the idea.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 12:22 am    Post subject: Reply with quote

I've tried those settings Neddy, and I started getting some errors:

Code:
distcc[7884] ERROR: compile CheckItem.c on 192.168.10.5/8,lzo,cpp failed
distcc[7884] (dcc_build_somewhere) Warning: remote compilation of 'CheckItem.c' failed, retrying locally
distcc[7884] Warning: failed to distribute CheckItem.c to 192.168.10.5/8,lzo,cpp, running locally instead


And you guys were right, using "pump" before emerge seems to invoke the distcc capability.
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 12:35 am    Post subject: Reply with quote

Just some more information, this is the command I used to create the ARM toolchain I am using:

Code:
crossdev -S -v -t armv6j-hardfloat-linux-gnueabi


It then compiled several packages without any errors, I'm assuming this would work? Or is there more to do?

*edit*

It apparently works:

Code:
shark bigun # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/cross-armv6j-hardfloat-linux-gnueabi/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=x86_64-pc-linux-gnu --target=armv6j-hardfloat-linux-gnueabi --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --disable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-poison-system-directories --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --with-sysroot=/usr/armv6j-hardfloat-linux-gnueabi --disable-bootstrap --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32443
Location: 56N 3W

PostPosted: Sun Nov 04, 2012 12:47 am    Post subject: Reply with quote

Bigun,

Its essential that you have the same version of gcc on the Pi as you do in the cross environment.

A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K

You can run the text mode distccmon-text on the Pi to see what distcc is doing.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 1:20 am    Post subject: Reply with quote

NeddySeagoon wrote:
Bigun,

Its essential that you have the same version of gcc on the Pi as you do in the cross environment.

A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K

You can run the text mode distccmon-text on the Pi to see what distcc is doing.


On my helper:

Code:
shark bigun # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/cross-armv6j-hardfloat-linux-gnueabi/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=x86_64-pc-linux-gnu --target=armv6j-hardfloat-linux-gnueabi --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --disable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-poison-system-directories --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --with-sysroot=/usr/armv6j-hardfloat-linux-gnueabi --disable-bootstrap --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7)



On the pi:

Code:
gmedia bin # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/sys-devel/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=armv6j-hardfloat-linux-gnueabi --build=armv6j-hardfloat-linux-gnueabi --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7)


It looks like the same version.

And yes, after the failed attempts it does compile normally on the pi. Let me try updating world and see what happens.
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 1:38 am    Post subject: Reply with quote

Nope, not 1 part of any package so far is being distributed. I've been monitoring both machines (the pi and helper), both remain empty.

Code:
armv6j-hardfloat-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I.. -DLOCALEDIR=\"/usr/share/locale\" -DSYSCONFDIR=\"/etc\"  -I/usr/include/ncursesw     -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -MT prompt.o -MD -MP -MF .deps/prompt.Tpo -c -o prompt.o prompt.c
distcc[28858] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
mv -f .deps/prompt.Tpo .deps/prompt.Po


I got this message at the end of the compile:

Code:
__________Warning: 2 pump-mode compilation(s) failed on server, but succeeded locally.
__________Distcc-pump was demoted to plain mode.  See the Distcc Discrepancy Symptoms section in the include_server(1) man page.
__________Shutting down distcc-pump include server
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 12:53 pm    Post subject: Reply with quote

No wait... it is working..... I think, the monitors on both machines however are showing jack squat.

I was watching htop on the helper machine while the pi was compiling stuff and watch distccd doing work (gkrellm also was showing the processing spikes).

So what's the deal here?

I'm running "distccmon-text 1" on both machines and neither are showing anything.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32443
Location: 56N 3W

PostPosted: Sun Nov 04, 2012 6:38 pm    Post subject: Reply with quote

Bigun,

distccmon-text 1 should only be run on the Pi. However thats only a part of the command.

You missed the elog at the end of the distcc install
Code:
        elog "Tips on using distcc with Gentoo can be found at"
        elog "http://www.gentoo.org/doc/en/distcc.xml"
        elog
        elog "How to use pump mode with Gentoo:"
        elog "# distcc-config --set-hosts \"foo,cpp,lzo bar,cpp,lzo baz,cpp,lzo\""
        elog "# pump emerge -u world"
        elog
        elog "To use the distccmon programs with Gentoo you should use this command:"
        elog "# DISTCC_DIR=\"${DISTCC_DIR}\" distccmon-text 5"

Change the "${DISTCC_DIR}\" to be wherever your distcc keeps its tracking files
For me its /var/tmp/portage/portage/.distcc so the command
Code:
DISTCC_DIR=/var/tmp/portage/portage/.distcc distccmon-text 5
prints out what distcc is doing every 5 seconds.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Sun Nov 04, 2012 8:05 pm    Post subject: Reply with quote

There it is!

Thanks Neddy, all is working!
Back to top
View user's profile Send private message
Bigun
Veteran
Veteran


Joined: 21 Sep 2003
Posts: 1974

PostPosted: Wed Nov 14, 2012 12:27 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Bigun,

Its essential that you have the same version of gcc on the Pi as you do in the cross environment.

A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K

You can run the text mode distccmon-text on the Pi to see what distcc is doing.


What is the process to do this? Compiling GCC is taking forever and my crossdev environment had it compiled in about 5 minutes.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32443
Location: 56N 3W

PostPosted: Wed Nov 14, 2012 7:14 pm    Post subject: Reply with quote

Bigun

On your cross build system you need gcc-4.6.3 as a cross compiler.
crossdev should be able to build that for you

Once your have the gcc-4.6.3 cross compiler in use (use gcc-config) you can use it to build a native gcc for the pi.
Code:
emerge-wrapper -init
armv6j-hardfloat-linux-gnueabi-emerge =gcc-4.6.3
there may be some odd -rX in the version too.

Before you build gcc for the Pi make sure you have FEATURES=buildpkg set in /usr/armv6j-hardfloat-linux-gnueabi/etc/portage/make.conf
This makes the build system save a binary tarball in your cross environment at PKGDIR=${ROOT}packages/
Its a really bad idea to mix host and target binaries in the same place.

When you have the tarball, move it to the equivelent location on the Pi. Preserve any directory structure you find in PKGDIR=${ROOT}packages/ down to gcc-4.6.3 or the next step will fail.
On the Pi emerge the binary tarball.
Code:
emerge -1K =gcc-4.6.3
-K (uppercase) says to use the binary package or fail. -k (lowercase) says to use the binary package if its found, if not build it.

As gcc will have been cross compiled, you will need the extra /lib/ld* symlink or it won't work.
You've been there before.
You will need to gcc-config on the Pi too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum