Automated emerge with distcc!

JeroenV · Posted: Thu Dec 19, 2002 11:39 am Post subject: Automated emerge with distcc!

Hi there,

I thought I'd share this new script with you (BEWARE: likely to contain bugs as it hasn't been thoroughly tested :!:

I only tested it with emerge -u world, so other command lines haven't been tested at all)
(Though bugs are not very likely to screw up your system :wink:

)

Ok, what it does:
when you emerge it switches on distcc usage by default, but when the build fails with distcc, it tries to build the failing package again without it. After that, distcc is again switched on for the remaining packages, until it fails... etc.
Very handy for your emerge -u world :!:

There are three ingredients:

1. /etc/env.d/90distcc:

JeroenV · Posted: Sat Dec 21, 2002 9:17 pm Post subject:

Sorry: two quite serious bugs :!:

The fix: script 3 should read like this:

dek · l33t Joined: 16 May 2002 Posts: 657 Location: Germany

Yes it really does something!

I started your script to compile qt on a almost fresh installed GCC3.2 system.
Unfortunately only the first package (libtiff) used distcc for the build. The second one in line (mysql) did not.

So i stopped the emerge after mysql was build. Next package was libmng.
I did "gentoo_distcc_emerge.sh libmng" and ended up in a loop. libmng compiles again and again, the fifth time right now, and it does not use distcc at all.
Any ideas whats going wrong here?
_________________
Computer games don't affect kids. I mean if Pacman affected us as kids, we'd all be running around in darkened rooms, munching pills and listening to repetitive music.

JeroenV · Posted: Sun Dec 22, 2002 8:00 pm Post subject:

Did you use the second version of script (3) :?:

Other than that you might check if script (2) changes file (1) the way it should: i.e. comment resp. uncomment line 1 and 2 of that file.
It may help to add some echo lines in script (3) to see exactly what commands are executed. That should provide some info on what's going wrong here.

Btw., kde-related stuff seems to be reluctant to use distcc? I'm not really sure, but I think the makefiles are constructed in a way that makes them ignore $CC and $CXX?

It's not perfect in the sense that I estimate about 70% of the packages of a gentoo system are actually able to use distcc, and of one such package about 20-40% of the total CPU-time would actually be distributed.

Anyway, an alternative would be mosix or cross-compiling, wich is significantly more work to get working.

Good luck :!:

EDIT:
I just noticed that older versions of portage (i.e. as found on stage3 installs) produce different outputs... It may be worth to change the TODO line in script (3) to

dek · l33t Joined: 16 May 2002 Posts: 657 Location: Germany

dek · l33t Joined: 16 May 2002 Posts: 657 Location: Germany

Oh wait, for some reason /etc/env.d/90distcc got overwritten. What happend here, the file was empty :?:

OK i give it another try.
I'm compiling a bunch of GTK/Gnome stuff right now. The first package used distcc and compiled fine, the second package in line does not use distcc. Same as before.

EDIT: Let me rephrase this:
The compiler output says distcc g++ ... for the second package, too. But the helper machine doesn't compile anything. Maybe i have to increase the "-j" settings. I use "-j3" right now. (i've got two machines)

I'll have to do more testing and report how it is going.
_________________
Computer games don't affect kids. I mean if Pacman affected us as kids, we'd all be running around in darkened rooms, munching pills and listening to repetitive music.

JeroenV · Posted: Sun Dec 22, 2002 11:26 pm Post subject:

dek · l33t Joined: 16 May 2002 Posts: 657 Location: Germany

JeroenV · Posted: Mon Dec 23, 2002 12:18 am Post subject:

Eh, one addendum: it's my script (2) for sure that kills 90distcc , there shouldn't be any other things touching that file... :oops:

I'll be investigating that... the sed line is probably not robust :?:

Anyway, I just started using sed :roll:

EDIT:
one more thing: apparently the emerge dependency order is not entirely fixed? So the assumption to "normally" emerge the failed package may lead to also normally emerging a dependency? Anyway, that might be a feature enhancement later...
_________________
Cheers

Jeroen
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
May The Source be with you!

dek · l33t Joined: 16 May 2002 Posts: 657 Location: Germany

There is still a bug in your script. Like i've already reported before, the last package always stays in a endless loop and gets compiled again and again.
Everything else works fine here. Some packages just don't make use of the helper machine. But i think this has nothing to do with your script. It is more likely to be concerned with distcc itself or the makefile of the given package.

I played a bit with different settings. So far i'm getting the best results with:

/etc/make.conf
MAKEOPTS="-j5"

/etc/env.d/90distcc
DISTCC_HOSTS="faster_machine slower_machine"

btw: Have you considered putting your scripts into "Documentation, Tips & Tricks"?
_________________
Computer games don't affect kids. I mean if Pacman affected us as kids, we'd all be running around in darkened rooms, munching pills and listening to repetitive music.

JeroenV · Posted: Mon Dec 23, 2002 5:08 pm Post subject:

Hi,

I'm unable to reproduce the endless loop you mention, I have no trouble with it... Maybe you could add some debugging lines concerning the TODO and REBUILD variables and check what exactly happens :?:

Is the order of the hosts in $DISTCC_HOSTS indeed of influence :?:

That's nice (maybe I should have RTFM'd) :wink:

Now I'm considering to write a more feature-rich script (maybe in python). The main objective is to really do cross-building: i.e. I will use my fastest machine as the main build host (and use distcc of course). It will work according to the following scheme:

1) use a customised set of USE and CFLAGS for the target host
2) get the list of packages to be emerged using emerge -p ....
3) get the first package from the list
4) double check whether emerge -p <first_package> indeed has no further dependencies
5) try a distcc-powered ebuild <first_package> package, if it fails, try without distcc again
6) repeat from (2) until done

and then (manually) do:

7 ) copy or mount the $PKGDIR on the target host
8 ) emerge -k ....

There is one (unsolvable) problem as I see it: compile time dependencies that are not satisfied on the build host will stop the build. The only 2 solutions:
1) build on the target host
2) first emerge normally on the build host and continue

EDIT:
I will put the next generation script in the Docs&Tips&Trics....
_________________
Cheers

Jeroen
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
May The Source be with you!

wayned · n00b Joined: 25 Dec 2002 Posts: 8

JeroenV · Posted: Wed Dec 25, 2002 2:16 pm Post subject:

french tony · Posted: Sat Dec 28, 2002 2:14 pm Post subject:

Hi, I'm trying to get distcc running.
This is my problem:

wayned · n00b Joined: 25 Dec 2002 Posts: 8

JeroenV · Posted: Sat Dec 28, 2002 10:46 pm Post subject:

Tony,

Also note that the gentoo_distcc_switch.sh is a rather ugly hack, even if you took the bug out. I have found my new hack to work better, see the link in my previous post.

Good luck :!:

_________________
Cheers

Jeroen
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
May The Source be with you!

kronenpj · n00b Joined: 06 Sep 2002 Posts: 3 Location: Orlando, FL

-switch is actually not too far off. Here's my update: