Share your distcc tips

[sarnold]

Okay, I think I mostly have it figured out now, but I'm sure some folks could add a few things too. Other than trying to figure out just which settings are current, and how it's different for Gentoo, the biggest problem I had was that distcc seems to stop using the extra hosts once in a while and falls back to local compiles only.

Here's what I learned in the last few days:

Follow the gentoo setup tips on zwelch's page here for the basic portage setup. Emerge distcc and ccache on all the machines you want to participate in the group builds (go ahead and emerge the latest 2.0.1 version of distcc, as it seems to work for me). Then add your DISTCC_HOSTS to make.conf, and set MAKEOPTS="-jN" where N is one more than twice the total number of CPUs used for distcc. And don't forget to add distcc and ccache to the FEATURES list.

You can check /etc/conf.d/distccd for reasonable settings, and add distccd to the default runlevel. Also, in order to get logging working (to a distcc log file instead of syslog) I added a /var/log/distcc directory (user=distcc, group=portage, rwxr-xr-x) and touched the distcc.log file (same ownership and permissions). Then I added the following to the /etc/conf.d/distccd file:

[rac]

[wick]

My $0.02:

- DISTCC_HOSTS should be ordered based on the particular machines' ability to compile. Remember that every idle machine that is running distccd provides the capability of running a "free" compile, regardless of its speed. The only time it makes a difference is when you have a make target that has very few source files. Make will wait for the slow machine before pressing on to the next target.

- In /etc/env.d, the CC= and CXX= options are set by the gcc environment options. I commented out the CC= anc CXX= lines in the environment options because I got tired of running unset CC and unset CXX at the command line prior to running emerge.

- Not all packages will build with distcc. It does not provide a clean unattended option for building a system, which is unfortunate because it would really facilitate bringing new boxes online quickly.

- As rac mentioned, there is no serious penalty for too many parallel compiles. However, eventually you'll find the breakeven point and start getting diminishing returns. My parallel option is set as -j15 now. I have 6 uniprocessor and 1 dual processor machines in my distcc build environment (which includes 2x vmware virtual machines running in Windows.) The theoretical maximum speed for my builds should be 7x as fast as a single machine compiling, but I don't get anywhere near that. It's typically around 5x.

[Beekster]

An additional question or two - seems like a good place for it:

What happens in the event the machines have different gcc or glibc versions? Or other package relevant to compiling? During the course of upgrading such packages on each machine, they will inevitably vary until they are all done. Can distcc be used to upgrade these packages?

[karrots]

[wick]

[mmealman]

What's the major differences between distcc 1.2 and 2.0?

[wick]

[Becks]

Ok, some questions then
Is there anyone here who has found an easy way to turn a machine into a distcc-host for compiling ? We have several (around 20) Athlon 1400 machines here running Windows NT. A knoppix cd with installed distcc would be great. Anyone there who has added this feature to knoppix and can explain how he did this ?

Alex

[Yarrick]

[versus]

[sn4ip3r]

If you want to run distcc on a machine without linux you don't have to use knoppix, because distcc works in cygwin. Ofcourse it means that you have to build a cross-compiler under cygwin which can be quite hard, but there is a guide at http://cvs.gentoo.org/~aiken/cygwin-cross/ and it works, mostly. I had to path gcc and I also had to manually remove a #define flag that enabled the use of mempcpy function (which didn't exist or work).

[skwelch]

Thanks for the info, this is way easy to setup and totally ends the pain of long compiles. Now I just need to sneek distccd onto other peoples workstations while they're at lunch

[versus]

when searching "distcc" on freshmeat I've found this link

http://www.warwick.ac.uk/~esvld/imageBoot/

Bootable diskette image which loads root filesystem via http/ftp.
Example root filesystem .tar.gz contains gcc 3.2.x + distcc !

gonna try this immediately

[Zadeh]

I've been building cygwin->linux cross compilers for awhile now to use distcc. There is a trivial patch that you need for gcc and libiberty which I'm trying to get backported to the 3.2 branch (already in 3.3) which is here:

http://gcc.gnu.org/ml/gcc/2003-04/msg00538.html

Basically all you need to do is
1) Copy all the /lib and /usr/lib files built by glibc and stuff it into a single directory, and copy that and /usr/include/* over to Cygwin. Modify the libc.so linker script to point to the new location of libc.so.6 and libc_nonshared.a in Cygwin.

2) Build binutils in Cygwin with --target=<whatever> as a configure setting. make -j<whatever> && make install

3) Prepend whatever prefix you're using (/usr/local by default) to your PATH. Pass --with-headers=<path to include dir> and --with-libs=<path to lib dir> when configuring gcc. It will copy the include and lib stuff to a subdir in your prefix. make -j<whatever> && make install

I've also built Linux->FreeBSD and Cygwin->FreeBSD cross compilers, but they were more difficult and I've still not worked out all the kinks yet. I've also had a lot of problems trying to get things built in win9x but had few issues with 2k/XP. If anyone can figure out or knows how to get a clean build under win9x I'd appreciate it.

[Zadeh]

Another thing I forgot to mention.. Instead of logging, I just start up distccd on the servers with --daemon --no-detach --log-stderr and watch them for a bit to make sure they're working. It will print out all thats going on to the "console" with a bunch of "got blahblah from IP# OK. Returned exit status 0" etc. After you confirm it works, you can keep it on that way or just set it and forget it.

[Beekster]

It would be nice to be able to use distcc early on in the Gentoo build process, say right after the bootstrap script, but an emerge -pe distcc gives quite a dependancy tree:

[Aiken]

[Zadeh]

The problem is that in several places the build looks for a builtin function, __mempcpy, but none exists on the Cygwin platform, so it needs to be just plain mempcpy.

It's needed if you want to build any cross compiler. See the link to the patch in my previous post.

[Aiken]

[Zadeh]

As far as I know. What version of cygwin? Also, have you checked that __mempcpy is in there? I've seen vanilla 3.2.2 fail for ->linux, ->arm, ->sh, ... with that issue.

[Becks]

I had several problems using distcc so far. Some packages can be compiled with disctc, others fail and have to be emerged locally. After setting /unsetting all those needed variables to en-/disable distcc i wrote a dirty little script which helps me.

To install it, do the following:
ACCEPT_KEYWORDS="~x86" emerge distcc (to get the latest version - 2.0.1 at the moment)

run distcc-config, add all the hosts which also run distccd, set the distcc-path and if you want - the distcc-log, but nothing else !

My variables are then set the following way:
CC=gcc
CXX=g++
DCCC_PATH=/usr/lib/distcc/bin
DISTCC_HOSTS='jeannie doom'
DISTCC_LOG=/var/log/distccd
DISTCC_VERBOSE=1
PATH='/usr/kde/3.1/bin:/bin:/usr/bin:/usr/local/bin.........

(The PATH doesn't have /usr/lib/distcc/bin as first entry !)

Don't change anything in make.conf. If you have enabled the following lines - set a # in front of them:
CC=distcc -> #CC=distcc
CXX=g++ -> #CXX=g++
FEATURES="distcc" ->#FEATURES="distcc"

if you have change the MAKEOPTS to something higher than your computer can handle - change it back (normally -j2 is set).

If you run an emerge now, distcc isn't used, everything should be compiled with gcc of your local machine.
Now take the following script:

[uziel]

Hello there,
I was just wondering in which way and order it would be best to combine distcc and ccache. As I've recently seen, they both work transparently through symlinks named like the compiler they are supposed to call. But that would require a certain "chain" of the directories in $PATH and the $CC / $CXX variables, e.g. $CC pointing to /usr/lib/distcc/bin/gcc and the first directory in $PATH containing a symlink called gcc was the ccache directory. I am therefore a little confused about how to set things up for compilations apart from portage. Which one should come first in the command line, and would it be suitable to simply do a "ccache distcc gcc blah.c" - which would require a modification of e.g. kernel makefiles?

Is there even a difference in function if the order of distcc / ccache is changed?

Sincerely
Chris
_________________
Those who promise their way of freedom and peace still do not understand the meaning of the word...

[tecknojunky]

I have just started experimenting with distcc and, so far, altough I still have some learning to do, i discovered that for faster machines it might slow the compile time.

It happenned to me when I added a 486 in the volonteers list. According to the distcc docs, it makes it sound like adding any kind of box is a bonux.

The 486 is simply just to slow for compiling for others. I cam to metaphore it to this: if you have to do the groceries at two different markets, asking a turtle to split the work may make things go slower because the time you finish your part, you realize you would have the turtle hasn't even arrived at the store .

So this is exactly what happenned to me. It seem there is a lower limit to which a fast machine can ask a slower machine to compile for it. If the machine is too slow, the faster box may wait so long for the slow box to finish that it would have had the time to compile it itself two times over (or more). So, my "fast" machine is a p3 500 and I must guess that for a whoping P4 2.4Ghz, a P1 66Mhz may in fact slow the compile. Or maybe with proper configuration... who knows? You? Tell me.

It is also true that it seem distcc stop delgating after a compile. My only solution, for now, is simply to start another shell, then it works again.

And, yes, some ./configure checks the gcc versions and the distcc wrapping messes that. To date, I got that with two packages, one being libxstl? or something, and the other was mplayer. In both case, it was the DISTCC_VERBOSE that needed to be commented.

Many tips I havent tried in this thread, thanks.
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax.

[Aiken]