Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Crazy compile mode
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 195

PostPosted: Tue Aug 17, 2010 4:59 pm    Post subject: Crazy compile mode Reply with quote

I have a fast multicore CPU and 4 GB RAM. Waiting after emerge -e world to finish it's a little bit frustrating.
The idea is to compile packages one by one, individually, without taking into account dependencies.
You will need app-shells/prll - http://prll.sourceforge.net/ witch will allow you to fully parallelize compile process
If you accomplish the following criteria:
1. an up to date Gentoo OS
2. You did an emerge -auvDN world
3. You already done revdep-rebuild
4. You want to recompile entire world set (and system set)
5. A fast multicore CPU and /var/tmp/portage mounted as tmpfs in RAM
you could try the following scripts.

compile_all.sh
Code:
#!/bin/bash
# Number of parallel emerge
JOBS=5
# First we update system
emerge --sync ; layman -S ; emerge -auvDN world
# Check revdep-rebuild and, if you use portage 2.2 preserved libraries
revdep-rebuild ; emerge @preserved-rebuild
# Full toolchain recompile
emerge --jobs=3 --oneshot portage linux-headers gcc-config mpfr binutils libtool dev-libs/mpc glibc gcc && env-update && source /etc/profile
# List of packages that need to compile
for f in `emerge -pe system | grep ebuild | gawk '{print $4;}' ` ; do echo -n $f" " ; done > system.txt
for f in `emerge -pe world  | grep ebuild | gawk '{print $4;}' ` ; do echo -n $f" " ; done > world.txt 
# Log file of every package compile status OK or FAIL
rm compile.log
touch compile.log
# Parallel compile of system and world
source /etc/profile.d/prll.sh
PRLL_NR_CPUS="$JOBS" prll -s 'compile_single.sh "$1"' `cat system.txt`
PRLL_NR_CPUS="$JOBS" prll -s 'compile_single.sh "$1"' `cat world.txt`


compile_single.sh
Code:
#!/bin/bash
# Maximum system load to permit a new emerge process
MAX_LOAD="7"
LOAD=`cat /proc/loadavg | gawk '{ print $1; }'`
# BASH does not allow floating numbers to be compared, so use bc to compare system load
COND=`echo "$LOAD > $MAX_LOAD" | bc`
# if system load is to high we wait to other emerge process to finish
while [ "$COND" -eq "1" ]; do
  sleep 5
  LOAD=`cat /proc/loadavg | gawk '{ print $1; }'`
  COND=`echo "$LOAD > $MAX_LOAD" | bc`
done

# Portage already have his log files, but if you need a new shiny one pick a location and a file name
LOG_FILE="/dev/null"
if (emerge --oneshot --nodeps ="$1" &>$LOG_FILE) ; then
  printf "%-50s: %s\n" "$1" "OK" >> compile.log ;
else
  printf "%-50s: %s\n" "$1" "FAIL" >> compile.log ;
fi


A lot of things can goes wrong if your system is not up to date and dependencies had not yet taken into account.
But if every package need just to be recompiled it will work.
It's a crazy compile mode so you better have a backup.
It's working fine in my case but it's just one case. This method it's not very well tested (I mean a big users base who use it), so use it on your own risk.
The big problem is when A package depends by B package but A it's compiled before B, so be very carefully and better meet the prerequisites.
I'm not responsible if you will break your system :D

Feel free to use the scripts, improve them, and, especially, let me know what you think about this method to recompile system and world sets.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2278
Location: UK

PostPosted: Tue Aug 17, 2010 6:56 pm    Post subject: Reply with quote

Got any statistics so we can compare?
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 195

PostPosted: Tue Aug 17, 2010 7:36 pm    Post subject: Reply with quote

My statistics aren't very scientific and improvements depends a lot by everyone USE flags.
In my case system set contain 551 packages and 382 of thats are compiling once at a time.
emerge -e system with MAKEOPTS="-j8 --load-average=11 and EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=7.9 --keep-going --with-bdeps=y --complete-graph" it's done now in 1/3 of initial time.
And emerge -e world it's done in ~60% of time than previous.
Average system load never fall under 4.5
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum