How-To Bootstrap and 'emerge system' with distcc

[securiteaze]

How-To Bootstrap with distcc
I have successfully bootstrapped a gentoo box from stage1-x86-1.4_rc1.
The following outlines the process used.
Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me

At the point during an install when you would normally run

Code: Select all

scripts/bootstrap.sh

Don't.
To use distcc while bootstrapping, do the following:

1. Emerge distcc to your system

   Code: Select all

   emerge distcc

2. Backup bootstrap.sh

   Code: Select all

   cp /usr/portage/scripts/bootstrap.sh /usr/portage/scripts/bootstrap.sh.orig

3. Edit bootstrap.sh to make it use distcc.

   Code: Select all

   nano -w /usr/portage/scripts/bootstrap.sh

   Find the following section in bootstrap.sh.

   Code: Select all

   cp /etc/make.conf /etc/make.conf.build
   export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
   export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
   export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
   export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"
   

4. Comment out the line :

   Code: Select all

   export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"
   

   Should now look like this :

   Code: Select all

   cp /etc/make.conf /etc/make.conf.build
   export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
   export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
   export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
   #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"
   

5. Add a new line above the one you just commented out.

   Code: Select all

   export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

   The -j12 specifies the number of threads to use; you may want to tweak this. (ie -j8 or -j32).
   The rest tells make to use distcc for compiling c and c++.
   
   Should now look like this :

   Code: Select all

   cp /etc/make.conf /etc/make.conf.build
   export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
   export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
   export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
   export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
   #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"
   

6. Add one more line :

   Code: Select all

   export DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv> <distccd-srv>"

   Replace <distccd-srv> with the hostname/IP of the boxes running distccd.
   This line tells distcc which hosts to send work and in which order.
   On a really slow box you may wish to leave out localhost or move it to last.
   
   Should now look similar to this:

   Code: Select all

   cp /etc/make.conf /etc/make.conf.build
   export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
   export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
   export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
   export DISTCC_HOSTS="localhost 10.1.1.25 10.1.1.50"
   export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
   #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"
   

7. Now, scroll the end of the script.*

   Code: Select all

   #make.conf has been overwritten, so we explicitly export our original settings
   export USE="$ORIGUSE bootstrap"
   emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
   #restore original make.conf
   cleanup 0
   

8. Find the following line and comment it out.*

   Code: Select all

   emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

   It should now look like

   Code: Select all

   #make.conf has been overwritten, so we explicitly export our original settings
   export USE="$ORIGUSE bootstrap"
   #emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
   #restore original make.conf
   cleanup 0
   

9. Now add the following lines above the one you commented out.*

   Code: Select all

   export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"
   emerge $myGLIBC || cleanup 1
   export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
   emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
   

   Should finally look like

   Code: Select all

   #make.conf has been overwritten, so we explicitly export our original settings
   export USE="$ORIGUSE bootstrap"
   export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"
   emerge $myGLIBC || cleanup 1
   export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
   emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
   #emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
   #restore original make.conf
   cleanup 0
   

10. Save bootstrap.sh then

    Code: Select all

    cd /usr/portage && scripts/bootstrap.sh
    

    Enjoy!

*Edited to fix a bug in emerging glibc 10/25
*Edited to change subject 10/27

Comments? Suggestions?

[securiteaze]

How-To emerge system with distcc
Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me

This method allows you to 'emerge system' using distcc.
A few packages have issues, but the workaround is included.

If you are continuing from a distcc bootstrap skip to step 2

1. Install distcc

   Code: Select all

   emerge sync ; emerge distcc

2. Tweak /etc/make.conf for distcc
   1. Backup make.conf

      Code: Select all

      cp /etc/make.conf /etc/make.conf.orig

   2. Edit /etc/make.conf to use distcc
      
      The following could be the entire content of make.conf

      Code: Select all

      #DISTCC_HOSTS tells distcc which hosts to send work
      DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv>"
      #Make sure to replace <distccd-srv> with the hostname/IP of the boxes running distccd
      
      #MAKEOPTS specifies commandline options when portage call the make command
      MAKEOPTS="-j16 CC='distcc' CXX='distcc'"
      #the -j16 tell make to spawn 16 threads, you may want to tweak this
      
      #If you don't know what the next lines do then see note below 
      CHOST="i686-pc-linux-gnu"
      CFLAGS="-march=pentium4 -O3 -pipe"
      CXXFLAGS="${CFLAGS}"
      

      Note: All about C flags
   3. Save make.conf
3. Now start the build

   Code: Select all

   emerge system

4. There will be some packages which fail to build(util-linux, bin86, procps, psmisc, netkit-base).
   For some reason some packages don't like the CC='distcc', CC='distcc gcc', CXX='distcc', or CXX='distcc g++'.
   Other packages (procps) don't like parallel makes (-j2).
   
   Edit /etc/make.conf and comment out the MAKEOPTS line.
   It should look like this

   Code: Select all

   #MAKEOPTS specifies commandline options when portage call the make command 
   #MAKEOPTS="-j16 CC='distcc' CXX='distcc'" 
   #the -j16 tell make to spawn 16 threads, you may want to tweak this 
   

5. Now build the finnicky packages

   Code: Select all

   emerge netkit-base procps psmisc

6. Edit /etc/make.conf and uncomment the MAKEOPTS line.

   Code: Select all

   #MAKEOPTS specifies commandline options when portage call the make command 
   MAKEOPTS="-j16 CC='distcc' CXX='distcc'" 
   #the -j16 tell make to spawn 16 threads, you may want to tweak this 
   

7. Continue the build

   Code: Select all

   emerge system

8. When 'emerge system' fails again repeat step 4
9. Build the last of the finnicky packages.

   Code: Select all

   emerge util-linux bin86

10. Repeat step 5 and 6. There shouldn't be any more build failures.
    
    Enjoy!

Comments? Suggestions?

[gentuse]

Good tip. I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here: http://lists.samba.org/pipermail/distcc ... 00308.html

[phong]

Thanks for the glibc workaround! I was going to write up this exact HOWTO but couldn't figure out how to get past glibc so I didn't bother (too bad there are other things that break unless you set CC="distcc" because they don't like a compiler that's two words). It's also nice to know it's an actual bug (non-recognition of the .S extension) and not just a general shortcoming that can't be fixed.

[BradN]

maybe you could make it run a script which then ran "distcc gcc" or something?

[BradB]

I followed this guide and had no problems installing. My workstation is an Athlon800 and the client is a Compaq Armada M300 (P2-300Mhz).
My only tips are
- to emerge packages without distcc and without changing /etc/make.conf you can

Code: Select all

MAKEOPTS="" emerge blah

- the distccd daemon will not run as root - took me ages to figure that one out!!

Cheers
Brad

[zojas]

ok, this is fun! but I've run into something strange. I'm bootstrapping on a 90 MHz pentium (this will be my iptables firewall and DNS server; until last night it was running redhat 7.2). my distcc_host contains only "raven", the hostname of my other machine, a 700 MHz athlon tbird.

Started the bootstrap around 9pm last night. At around 9:30 this morning was the last time the distccd on raven was connected to. So I'm thinking it's either done or it failed. when I got home just now I see that not only is it still running but it's invoking gcj to compile java code!! what package is that? this is the bootstrap.sh step still. so apparently it's been putzing with java for 8 hours, which of course all has to compile locally.

[zhenlin]

gcj is java, and is probably from gcc compiling it's own class files. USE="-java"

[Jeld]

I have followed this guide, to the letter, and some of the stuff has compiled very fast ( Celeron 533 as client Athlon 1800 XP server ). Gcc though doesn't seem to use distcc. At some point into the compilation it switches to something it calles xgcc and compiles using that.

[phong]

Unfortunately, that is a fact of life when compiling gcc. First it compiles itself using the host compiler (in this case, gcc via. distcc) using very conservative options. Then using that compiled version, it compiles itself again using optimizations. This part can't be done with distcc because it needs to use its freshly compiled copy. Then it compiles itself for a third and final time using the new optimized version. Again, this can't be done with distcc. Finally, the two final versions are compared. If they're identical, it considers itself to be working and installs. If not, it knows something bad happened along the way. Unfortunately, compling three times is slow, and more noticably when using distcc since that only works for the first one.

[Jeld]

Sorry for my last comment, I already found this information in a different thread. currently I am doing emerge system, and some packages seem to be not distcc friendly. So far I found that bash will use gcc no matter what MAKEOPT says.

Edit: I will start a collection of packages that do not use MAKEOPTS

bash-2.05b-r3
groff-1.18.1
perl-5.8.0-r9
db-3.2.9
expat-1.95.5-r1
ttmkfdir-3.0.4
pam-0.75-r11

XFree is compiling now, so my collection will probably have to stop here ( will let emerge run overnight ). A couple of things worth mentioning abuot speeding up the compile.

1. If you are trying to install on a slower system and have one fast system available you can try to partition the drive, mount partitions in /mnt/gentoo export /mnt/gentoo over nfs and mount on the fast system. Then you chroot and build. I cuoldn't get the nfsd running off the live CD, but then again, my CD is a bit outdated ( 1.4_rc1 I believe ).
2. On most systems ( even the slower ones ) hard drive access times are much slower then compile times. Therefore setting MAKEOPTS to -j8 or something similar may speed up the compile even on a single CPU system due to the fact that several source files are read at once and then compiled according to the kernel schedule, instead of having to wait for multiple disk accesses.
3. In the same spirit, during compiles gcc writes temp files into /tmp doing
mount -t tmpfs tmpfs /mnt/gentoo/tmp
imprives compile speeds dramatically on both my Athlon 1800 XP and Celeron 533.

Good luck.

[AlterEgo]

3. In the same spirit, during compiles gcc writes temp files into /tmp doing
mount -t tmpfs tmpfs /mnt/gentoo/tmp
imprives compile speeds dramatically on both my Athlon 1800 XP and Celeron 533.

Jeld, please explain this one. I don't understand the gain in this

[Jeld]

I am not sure, but logicaly speaking, disk writes are time consuming, and since the temp object files are going to be needed by gcc right away teh write operation have to be completed then and there ( i.e. the write is not really buffered ). So instead of writing temp files to the disk you are writing them to a RAM disk ( tempfs in this case ). Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster. At least that is how I explain this, I might be wrong. It does speed things up quite a bit even on my Celeron 533 128 MB RAM. Also some other operations require temp files such as gzip, groff etc.

[ekoontz]

Just to follow up on the suggestion that BradN made above.
(also mentioned by Martin Pool here : http://lists.samba.org/pipermail/distcc ... 00308.html


I tried having :

Code: Select all

MAKEOPTS="-j3 CC=distcc CXX=distcc g++"

But this did not work for some packages that seemed to want CXX to be one token. So, I wrote a wrapper script which I installed as /usr/bin/distcc-c++ :

Code: Select all

#!/bin/sh
distcc c++ $*

Then I put in my make.conf :

Code: Select all

MAKEOPTS="-j3 CC=distcc CXX=distcc-c++"

Works well now!

[Donovan]

Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster.

Exactly! Using tmpfs means it's being held in RAM, and no matter how fast your hard drives are, they'll never be as fast as pure RAM. Since it's not an actual RAM disk, we don't have to worry about allocating RAM to something that won't be used. If tmp is blank, no memory is used, as it's allocated on the fly. Additionally, this frees up the HD from having to handle the temp files. It's great! (You need TMPFS support in your kernel of course)

The Amigas had a RAM disk like this back in the 80s. Of course, back then, the GUI and programs ran on a machine with 1/2MB of memory (and still do). What are the requirements for KDE again?

[Normie]

I've got a slow Gentoo system (400 mhz p2, 320mb ram), and a fairly speedy XP system (1.4ghz athlon, 512mb ram). I originally wanted to install gentoo on the XP system and boot it with a grub floppy for some good ole' distccd whoring (;)), but for really stupid reason's I can't (and won't bother to) explain here, I can't repartition. So I figure I'll just use vmware when I need distccd assistance. Does anyone know how much my performance can expect to be cut (as compared to a "true" gentoo install)? 50%? 60%? What?

(edit) this is OT, I know, but if anyone knows a better way than vmware to install a barebones, made-for-distccd gentoo system without repartitioning, please PM me.(/edit)

[elpierco]

I started emerging the system and I see using netstat and top u distcc that it is distributing work to my other machine(an Athlon XP). I am trying to build the system on a P4 2.8C...the emerge runs for a while then I get this

checking for i686-pc-linux-gnu-g++... gcc
checking whether we are using the GNU C++ compiler... no
checking whether gcc accepts -g... no
checking dependency style of gcc... none
checking how to run the C++ preprocessor... /lib/cpp
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.
configure: error: /bin/sh './configure' failed for autoconf-lib-link

!!! ERROR: sys-devel/gettext-0.12.1 failed.
!!! Function econf, Line 362, Exitcode 1
!!! econf failed

Should I try to emerge that without distcc? sys-devel/gettext-0.12.1?

here is what my make.conf looks like

# These settings were set by the catalyst build script that automatically built$
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-O3 -march=i686 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="-qt -kde"
FEATURES="distcc"
DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
MAKEOPTS="-j5"
CC="distcc gcc"
CXX="distcc g++"

[elpierco]

So I am going to use the minimal install disk and try this one more time...

Just to have some confirmation on the machine (localhost) that will be building the stage 1 and distributing jobs to others. Does it matter if CC="distcc" is set on the other boxes?

Does it matter if this is on a seperate line or should it be included with MAKEOPTS i.e. MAKEOPTS="-j7 CC=distcc"

Does anyone have any ideas what that error might be? This is a brand new system components wise. I will be using a P4 2.8GHz and an Athlon 1800XP.