View previous topic :: View next topic |
Author |
Message |
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sat Nov 03, 2012 11:37 pm Post subject: Need help getting distcc working (solved) |
|
|
I'm trying to get distcc working with a strict client/server environment.
Here is the configuration I have so far:
Server Configuration
IP Address:
/etc/conf.d/distccd:
Code: | # /etc/conf.d/distccd: config file for /etc/init.d/distccd
DISTCCD_OPTS=""
# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"
# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
# set this option to run distccd with extra parameters
# Default port is 3632. For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
# SECURITY NOTICE:
# It is HIGHLY recomended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example: --allow 192.168.0.0/24
# example: --allow 192.168.0.5 --allow 192.168.0.150
# example: --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.10.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.10.5"
# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15" |
/etc/make.conf:
Code: | # These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-march=core2 -O2 -pipe"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
USE="X a52 aac alsa apng bidi bsf cdda cdio dbus dts dvd fat ffmpeg flac gif
gimp gnome gtk gudev hwdb jpeg libkms live lua mad matroska minizip mjpeg
mmx mp3 mp4 mpeg mplayer nvidia ogg opengl png quicktime sse sse2 stream
theora tiff truetype udev vcd vlm vorbis wav wma x264 xorg xv xvid"
MAKEOPTS="-j3"
INPUT_DEVICES="evdev"
VIDEO_CARDS="nvidia"
PORTDIR_OVERLAY="/usr/local/portage"
FEATURES="distcc" |
/etc/distcc/hosts:
/etc/init.d/distccd is running
Notes:
This machine already has the toolchain build for the architecture I'm building for. I've been using the distcc monitor and haven't seen any activity.
Client Configuration
IP Address:
/etc/conf.d/distccd:
Code: | # /etc/conf.d/distccd: config file for /etc/init.d/distccd
DISTCCD_OPTS=""
# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"
# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
# set this option to run distccd with extra parameters
# Default port is 3632. For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
# SECURITY NOTICE:
# It is HIGHLY recomended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example: --allow 192.168.0.0/24
# example: --allow 192.168.0.5 --allow 192.168.0.150
# example: --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"
# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15" |
/etc/make.conf:
Code: | # These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="armv6j-hardfloat-linux-gnueabi"
MAKEOPTS="-j5"
FEATURES="parallel-fetch buildpkg distcc"
#GENTOO_MIRRORS="http://intelminer.com/gentoo"
#SYNC="rsync://intelminer.com/gentoo-portage"
ACCEPT_LICENSE="*"
EMERGE_DEFAULT_OPTS="--quiet-build=n"
PORTAGE_NICENESS="15"
PKGDIR="/var/binary"
USE="gif jpeg mad mp3 png quicktime theora x264-static xv xvid" |
/etc/distcc/hosts:
/etc/init.d/distccd is running
Notes:
This is a raspberry pi. When I run emerge, it does compile whatever I ask it to, but it is obvious that it's doing it itself and not using distcc.
What am I missing here? _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim
Last edited by Bigun on Sun Nov 04, 2012 8:06 pm; edited 1 time in total |
|
Back to top |
|
|
Jaglover Watchman
Joined: 29 May 2005 Posts: 8291 Location: Saint Amant, Acadiana
|
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sat Nov 03, 2012 11:57 pm Post subject: |
|
|
Pump mode? I looked into that briefly and I think that may be an alternative to distcc, correct? _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
Jaglover Watchman
Joined: 29 May 2005 Posts: 8291 Location: Saint Amant, Acadiana
|
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54099 Location: 56N 3W
|
Posted: Sun Nov 04, 2012 12:14 am Post subject: |
|
|
Bigun,
Pump mode allows more than compiles to be offloaded.
On my Pi /etc/distcc/hosts contains Code: | 192.168.100.20/8,lzo,cpp | which tells it to send a maximum of 8 jobs to my Phenom II 1090.
lzo is for compression, which pump mode requires and cpp allows the use of pump mode.
make.conf on the Pi contains
Code: | # -j8 for distcc, -j1 or -j2 when there are no helpers
MAKEOPTS="-j2" |
The helper is set up to listem on 192.168.0.0/16 because its on several subnets
Code: | pump emerge kde-meta | on the Pi sends most of the work to the cross compiler on the AMD box.
No - you can't run KDE on a Pi but you get the idea. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 12:22 am Post subject: |
|
|
I've tried those settings Neddy, and I started getting some errors:
Code: | distcc[7884] ERROR: compile CheckItem.c on 192.168.10.5/8,lzo,cpp failed
distcc[7884] (dcc_build_somewhere) Warning: remote compilation of 'CheckItem.c' failed, retrying locally
distcc[7884] Warning: failed to distribute CheckItem.c to 192.168.10.5/8,lzo,cpp, running locally instead |
And you guys were right, using "pump" before emerge seems to invoke the distcc capability. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 12:35 am Post subject: |
|
|
Just some more information, this is the command I used to create the ARM toolchain I am using:
Code: | crossdev -S -v -t armv6j-hardfloat-linux-gnueabi |
It then compiled several packages without any errors, I'm assuming this would work? Or is there more to do?
*edit*
It apparently works:
Code: | shark bigun # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/cross-armv6j-hardfloat-linux-gnueabi/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=x86_64-pc-linux-gnu --target=armv6j-hardfloat-linux-gnueabi --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --disable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-poison-system-directories --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --with-sysroot=/usr/armv6j-hardfloat-linux-gnueabi --disable-bootstrap --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7) |
_________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54099 Location: 56N 3W
|
Posted: Sun Nov 04, 2012 12:47 am Post subject: |
|
|
Bigun,
Its essential that you have the same version of gcc on the Pi as you do in the cross environment.
A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K
You can run the text mode distccmon-text on the Pi to see what distcc is doing. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 1:20 am Post subject: |
|
|
NeddySeagoon wrote: | Bigun,
Its essential that you have the same version of gcc on the Pi as you do in the cross environment.
A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K
You can run the text mode distccmon-text on the Pi to see what distcc is doing. |
On my helper:
Code: | shark bigun # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/cross-armv6j-hardfloat-linux-gnueabi/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=x86_64-pc-linux-gnu --target=armv6j-hardfloat-linux-gnueabi --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --disable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-poison-system-directories --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --with-sysroot=/usr/armv6j-hardfloat-linux-gnueabi --disable-bootstrap --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7)
|
On the pi:
Code: | gmedia bin # armv6j-hardfloat-linux-gnueabi-gcc --ver
Using built-in specs.
COLLECT_GCC=/usr/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4/armv6j-hardfloat-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/lto-wrapper
Target: armv6j-hardfloat-linux-gnueabi
Configured with: /var/tmp/portage/sys-devel/gcc-4.5.4/work/gcc-4.5.4/configure --prefix=/usr --bindir=/usr/armv6j-hardfloat-linux-gnueabi/gcc-bin/4.5.4 --includedir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include --datadir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4 --mandir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/man --infodir=/usr/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/info --with-gxx-include-dir=/usr/lib/gcc/armv6j-hardfloat-linux-gnueabi/4.5.4/include/g++-v4 --host=armv6j-hardfloat-linux-gnueabi --build=armv6j-hardfloat-linux-gnueabi --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --with-float=hard --enable-nls --without-included-gettext --with-system-zlib --enable-obsolete --disable-werror --enable-secureplt --disable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/armv6j-hardfloat-linux-gnueabi/4.5.4/python --enable-checking=release --disable-libgcj --with-arch=armv6j --with-float=hard --with-fpu=vfp --enable-languages=c,c++,fortran --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.4 p1.0, pie-0.4.7'
Thread model: posix
gcc version 4.5.4 (Gentoo 4.5.4 p1.0, pie-0.4.7)
|
It looks like the same version.
And yes, after the failed attempts it does compile normally on the pi. Let me try updating world and see what happens. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 1:38 am Post subject: |
|
|
Nope, not 1 part of any package so far is being distributed. I've been monitoring both machines (the pi and helper), both remain empty.
Code: | armv6j-hardfloat-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I.. -DLOCALEDIR=\"/usr/share/locale\" -DSYSCONFDIR=\"/etc\" -I/usr/include/ncursesw -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -MT prompt.o -MD -MP -MF .deps/prompt.Tpo -c -o prompt.o prompt.c
distcc[28858] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
mv -f .deps/prompt.Tpo .deps/prompt.Po |
I got this message at the end of the compile:
Code: | __________Warning: 2 pump-mode compilation(s) failed on server, but succeeded locally.
__________Distcc-pump was demoted to plain mode. See the Distcc Discrepancy Symptoms section in the include_server(1) man page.
__________Shutting down distcc-pump include server |
_________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 12:53 pm Post subject: |
|
|
No wait... it is working..... I think, the monitors on both machines however are showing jack squat.
I was watching htop on the helper machine while the pi was compiling stuff and watch distccd doing work (gkrellm also was showing the processing spikes).
So what's the deal here?
I'm running "distccmon-text 1" on both machines and neither are showing anything. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54099 Location: 56N 3W
|
Posted: Sun Nov 04, 2012 6:38 pm Post subject: |
|
|
Bigun,
distccmon-text 1 should only be run on the Pi. However thats only a part of the command.
You missed the elog at the end of the distcc install
Code: | elog "Tips on using distcc with Gentoo can be found at"
elog "http://www.gentoo.org/doc/en/distcc.xml"
elog
elog "How to use pump mode with Gentoo:"
elog "# distcc-config --set-hosts \"foo,cpp,lzo bar,cpp,lzo baz,cpp,lzo\""
elog "# pump emerge -u world"
elog
elog "To use the distccmon programs with Gentoo you should use this command:"
elog "# DISTCC_DIR=\"${DISTCC_DIR}\" distccmon-text 5" |
Change the "${DISTCC_DIR}\" to be wherever your distcc keeps its tracking files
For me its /var/tmp/portage/portage/.distcc so the command
Code: | DISTCC_DIR=/var/tmp/portage/portage/.distcc distccmon-text 5 | prints out what distcc is doing every 5 seconds. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Sun Nov 04, 2012 8:05 pm Post subject: |
|
|
There it is!
Thanks Neddy, all is working! _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
Bigun Advocate
Joined: 21 Sep 2003 Posts: 2196
|
Posted: Wed Nov 14, 2012 12:27 pm Post subject: |
|
|
NeddySeagoon wrote: | Bigun,
Its essential that you have the same version of gcc on the Pi as you do in the cross environment.
A few failures are normal as long as they recompile locally normally.
You may also build binaries in your cross environment that you move to the Pi and install with emerge -K
You can run the text mode distccmon-text on the Pi to see what distcc is doing. |
What is the process to do this? Compiling GCC is taking forever and my crossdev environment had it compiled in about 5 minutes. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54099 Location: 56N 3W
|
Posted: Wed Nov 14, 2012 7:14 pm Post subject: |
|
|
Bigun
On your cross build system you need gcc-4.6.3 as a cross compiler.
crossdev should be able to build that for you
Once your have the gcc-4.6.3 cross compiler in use (use gcc-config) you can use it to build a native gcc for the pi.
Code: | emerge-wrapper -init
armv6j-hardfloat-linux-gnueabi-emerge =gcc-4.6.3 | there may be some odd -rX in the version too.
Before you build gcc for the Pi make sure you have FEATURES=buildpkg set in /usr/armv6j-hardfloat-linux-gnueabi/etc/portage/make.conf
This makes the build system save a binary tarball in your cross environment at PKGDIR=${ROOT}packages/
Its a really bad idea to mix host and target binaries in the same place.
When you have the tarball, move it to the equivelent location on the Pi. Preserve any directory structure you find in PKGDIR=${ROOT}packages/ down to gcc-4.6.3 or the next step will fail.
On the Pi emerge the binary tarball.
Code: | emerge -1K =gcc-4.6.3 | -K (uppercase) says to use the binary package or fail. -k (lowercase) says to use the binary package if its found, if not build it.
As gcc will have been cross compiled, you will need the extra /lib/ld* symlink or it won't work.
You've been there before.
You will need to gcc-config on the Pi too. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
|