| View previous topic :: View next topic |
| Author |
Message |
costel78
Guru
Joined: 20 Apr 2007
Posts: 402
|
 Posted: Fri Aug 19, 2011 7:59 am Post subject: Fast world recompile |
 |
|
Sometimes, as every Gentoo user, I have to do a full world recompile. And I have an Intel Core i7 860 CPU.
Looking at it how cores just stand, increasing the time I have to wait until recompilation it's done, make me impatient.
So I read manuals of portage and make.conf I run some test. That was almost two months ago.
Prerequisites:
1. All packages are up-to-date (emerge -auvDN world);
2. No link-inconsistency (revedep-rebuild and emerge @module-rebuild);
3. No other various portage error: (emaint --fix all; eix-update ; eix-test-obsolete);
4. portage-2.2.x - I haven't tested the setup with portage 2.1.x . Anyway, 2.1 version does not support some of vital required feature for this setup.
The way I do a full system and world recompile is:
1. Recompile toolchain in standard way;
2. Obtain a list with all system packages;
3. Obtain a list with all world packages;
4. Compile the two list (one at a time) using --nodeps --oneshoot using aggressive FEATURES to speed up parallelization.
Relevant entries in make.conf:
| Code: | COLLISION_IGNORE="/bin /lib /sbin /usr /var"
EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y --complete-graph"
FEATURES="assume-digests candy collision-protect -ebuild-locks distlocks fail-clean fixpackages news nodoc noinfo parallel-fetch parallel-install -ebuild-locks protect-owned sandbox strict -test unmerge-orphans" |
Increasing number of parallel package builds over 3 (--jobs=3) are pointless, well, unless you have a hexacore AMD CPU I suppose.
Modify load-average value according to your CPU.
Take a look at fail-clean parallel-fetch parallel-install -ebuild-locks and sandbox portage features. Here is the half of the magic 
The other half:
recompile.sh
| Code: | emerge --oneshot portage linux-headers gcc-config mpfr binutils libtool dev-libs/mpc dev-libs/ppl dev-libs/cloog-ppl glibc gcc && env-update && source /etc/profile
for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ;
for f in `emerge -pve world | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > world_list.txt ;
emerge --nodeps --oneshot `cat system_list.txt` ;
emerge --nodeps --oneshot `cat world_list.txt` |
mpc is for newer gcc, ppl and cloog-ppl are for graphite support. Leave them or remove them, depending of your gcc version/use-flags.
MAKEOPTS it's also at your discretion.
Enjoy fastes world recompile.
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
 |
audiodef
Watchman
Joined: 06 Jul 2005
Posts: 6639
Location: The soundosphere
|
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 402
|
| Posted: Fri Aug 19, 2011 4:46 pm Post subject: |
|
|
Yes, I understand the irony, but it's seem like waste of CPU power to compile only one package at a time, with CPU between 5%-35%, especially when you spent a fortune to buy it.
Regarding MAKEOPTS, it control only one package. Many packages force "-j1" or are two small to benefit from parallel compile. Here is where EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=" appear.
For comparation:
emerge -e system took 285% as | Code: | | for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ; emerge --nodeps --oneshot `cat system_list.txt` ; |
because many packages in system are compiling only one at a time therefor time required to install them count on total time.
Emerge -e world took ~ 11 hours. Using the above procedure took only 5:36 hours.
pkgcore developers (maybe sometime in the future this will become portage3) takes a similar approach.
I only want to share the info as pkgcore it's still immature. Someones may find it usefully, someones futile. It's ok 
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54234
Location: 56N 3W
|
| Posted: Fri Aug 19, 2011 5:54 pm Post subject: |
|
|
costel78,
Its very rare that you need to rebuild your entire install. I think the last time it was actually requied was arounf gcc-3.4 when the ABI changed.
--jobs= makes a pretty good job of parallel building as it respects dependencies too. That might be important.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
dwbowyer
Apprentice
Joined: 18 Apr 2008
Posts: 155
|
| Posted: Fri Aug 19, 2011 8:30 pm Post subject: |
|
|
All that is really needed is
| /etc/make.conf: |
EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y
|
Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that. |
|
| Back to top |
|
|
audiodef
Watchman
Joined: 06 Jul 2005
Posts: 6639
Location: The soundosphere
|
|
| Back to top |
|
|
dwbowyer
Apprentice
Joined: 18 Apr 2008
Posts: 155
|
| Posted: Fri Aug 19, 2011 8:49 pm Post subject: |
|
|
Maybe he is like me... 26GB / is 57% full. and that with /home, /usr/portage, and /usr/share all on separate partitions.
I love my games. |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 402
|
| Posted: Sat Aug 20, 2011 6:12 am Post subject: |
|
|
| Quote: | | Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that. |
Well, it's seem that my dependencies list prevent a good parallelization - a long list of only one package at a time... 
I didn't recompile every package in the portage,  but I use some , hmmm... not widely used.
I'm glad that it's only me with slow emerge -e @system @world. Particular situations, particular solutions
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
Hypnos
Advocate
Joined: 18 Jul 2002
Posts: 2889
Location: Omnipresent
|
| Posted: Sat Aug 20, 2011 6:23 am Post subject: Re: Fast world recompile |
|
|
| costel78 wrote: | | Sometimes, as every Gentoo user, I have to do a full world recompile. |
FWIW, I have used Gentoo since 2002, and never done a world recompile. Even with the gcc-3.4 C++ ABI change, I just recompiled the C++ programs -- not very many, since I used GNOME at the time.
_________________
Personal overlay | Simple backup scheme |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
