Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Fast world recompile
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 203

PostPosted: Fri Aug 19, 2011 7:59 am    Post subject: Fast world recompile Reply with quote

Sometimes, as every Gentoo user, I have to do a full world recompile. And I have an Intel Core i7 860 CPU.
Looking at it how cores just stand, increasing the time I have to wait until recompilation it's done, make me impatient.
So I read manuals of portage and make.conf I run some test. That was almost two months ago.

Prerequisites:
1. All packages are up-to-date (emerge -auvDN world);
2. No link-inconsistency (revedep-rebuild and emerge @module-rebuild);
3. No other various portage error: (emaint --fix all; eix-update ; eix-test-obsolete);
4. portage-2.2.x - I haven't tested the setup with portage 2.1.x . Anyway, 2.1 version does not support some of vital required feature for this setup.

The way I do a full system and world recompile is:
1. Recompile toolchain in standard way;
2. Obtain a list with all system packages;
3. Obtain a list with all world packages;
4. Compile the two list (one at a time) using --nodeps --oneshoot using aggressive FEATURES to speed up parallelization.

Relevant entries in make.conf:
Code:
COLLISION_IGNORE="/bin /lib /sbin /usr /var"
EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y --complete-graph"
FEATURES="assume-digests candy collision-protect -ebuild-locks distlocks fail-clean fixpackages news nodoc noinfo parallel-fetch parallel-install -ebuild-locks protect-owned sandbox strict -test unmerge-orphans"


Increasing number of parallel package builds over 3 (--jobs=3) are pointless, well, unless you have a hexacore AMD CPU I suppose.
Modify load-average value according to your CPU.
Take a look at fail-clean parallel-fetch parallel-install -ebuild-locks and sandbox portage features. Here is the half of the magic :D

The other half:
recompile.sh
Code:
emerge --oneshot portage linux-headers gcc-config mpfr binutils libtool dev-libs/mpc dev-libs/ppl dev-libs/cloog-ppl glibc gcc && env-update && source /etc/profile
for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ;
for f in `emerge -pve world | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > world_list.txt ;
emerge --nodeps --oneshot `cat system_list.txt` ;
emerge --nodeps --oneshot `cat world_list.txt`

mpc is for newer gcc, ppl and cloog-ppl are for graphite support. Leave them or remove them, depending of your gcc version/use-flags.
MAKEOPTS it's also at your discretion.

Enjoy fastes world recompile.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5282

PostPosted: Fri Aug 19, 2011 1:44 pm    Post subject: Reply with quote

Whatever happened to using MAKEOPTS and emerge @system @world? 8O
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 203

PostPosted: Fri Aug 19, 2011 4:46 pm    Post subject: Reply with quote

Yes, I understand the irony, but it's seem like waste of CPU power to compile only one package at a time, with CPU between 5%-35%, especially when you spent a fortune to buy it.
Regarding MAKEOPTS, it control only one package. Many packages force "-j1" or are two small to benefit from parallel compile. Here is where EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=" appear.
For comparation:
emerge -e system took 285% as
Code:
for f in `emerge -pve system | grep ebuild | gawk '{print $4;}'` ; do echo -n " ="$f ; done > system_list.txt ; emerge --nodeps --oneshot `cat system_list.txt` ;
because many packages in system are compiling only one at a time therefor time required to install them count on total time.
Emerge -e world took ~ 11 hours. Using the above procedure took only 5:36 hours.

pkgcore developers (maybe sometime in the future this will become portage3) takes a similar approach.
I only want to share the info as pkgcore it's still immature. Someones may find it usefully, someones futile. It's ok :)
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32112
Location: 56N 3W

PostPosted: Fri Aug 19, 2011 5:54 pm    Post subject: Reply with quote

costel78,

Its very rare that you need to rebuild your entire install. I think the last time it was actually requied was arounf gcc-3.4 when the ABI changed.

--jobs= makes a pretty good job of parallel building as it respects dependencies too. That might be important.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
dwbowyer
Apprentice
Apprentice


Joined: 18 Apr 2008
Posts: 154

PostPosted: Fri Aug 19, 2011 8:30 pm    Post subject: Reply with quote

All that is really needed is
/etc/make.conf:

EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=8.1 --keep-going --with-bdeps=y


Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that.
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5282

PostPosted: Fri Aug 19, 2011 8:35 pm    Post subject: Reply with quote

My Phenom II x6 system doesn't take more than a few hours - if that - to emerge world, without any special tweaking. Did you emerge every package in Portage? :P 8O
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
dwbowyer
Apprentice
Apprentice


Joined: 18 Apr 2008
Posts: 154

PostPosted: Fri Aug 19, 2011 8:49 pm    Post subject: Reply with quote

Maybe he is like me... 26GB / is 57% full. and that with /home, /usr/portage, and /usr/share all on separate partitions.

I love my games.
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 203

PostPosted: Sat Aug 20, 2011 6:12 am    Post subject: Reply with quote

Quote:
Just the "--jobs --load-average" parallelizes multiple packages, and caps the load average so as not to swamp they given system. I've been using that for 2 years now on my AMD64x2 (since portage 1.9 I think). Everything else suggested seems to be about build system and world packages in the right order, and there are many solutions for that.


Well, it's seem that my dependencies list prevent a good parallelization - a long list of only one package at a time... :(
I didn't recompile every package in the portage, :lol: but I use some , hmmm... not widely used.
I'm glad that it's only me with slow emerge -e @system @world. Particular situations, particular solutions :)
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2868
Location: Omnipresent

PostPosted: Sat Aug 20, 2011 6:23 am    Post subject: Re: Fast world recompile Reply with quote

costel78 wrote:
Sometimes, as every Gentoo user, I have to do a full world recompile.

FWIW, I have used Gentoo since 2002, and never done a world recompile. Even with the gcc-3.4 C++ ABI change, I just recompiled the C++ programs -- not very many, since I used GNOME at the time.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum