| View previous topic :: View next topic |
| Author |
Message |
costel78
Guru
Joined: 20 Apr 2007
Posts: 402
|
 Posted: Tue Aug 17, 2010 4:59 pm Post subject: Crazy compile mode |
 |
|
I have a fast multicore CPU and 4 GB RAM. Waiting after emerge -e world to finish it's a little bit frustrating.
The idea is to compile packages one by one, individually, without taking into account dependencies.
You will need app-shells/prll - http://prll.sourceforge.net/ witch will allow you to fully parallelize compile process
If you accomplish the following criteria:
1. an up to date Gentoo OS
2. You did an emerge -auvDN world
3. You already done revdep-rebuild
4. You want to recompile entire world set (and system set)
5. A fast multicore CPU and /var/tmp/portage mounted as tmpfs in RAM
you could try the following scripts.
compile_all.sh
| Code: | #!/bin/bash
# Number of parallel emerge
JOBS=5
# First we update system
emerge --sync ; layman -S ; emerge -auvDN world
# Check revdep-rebuild and, if you use portage 2.2 preserved libraries
revdep-rebuild ; emerge @preserved-rebuild
# Full toolchain recompile
emerge --jobs=3 --oneshot portage linux-headers gcc-config mpfr binutils libtool dev-libs/mpc glibc gcc && env-update && source /etc/profile
# List of packages that need to compile
for f in `emerge -pe system | grep ebuild | gawk '{print $4;}' ` ; do echo -n $f" " ; done > system.txt
for f in `emerge -pe world | grep ebuild | gawk '{print $4;}' ` ; do echo -n $f" " ; done > world.txt
# Log file of every package compile status OK or FAIL
rm compile.log
touch compile.log
# Parallel compile of system and world
source /etc/profile.d/prll.sh
PRLL_NR_CPUS="$JOBS" prll -s 'compile_single.sh "$1"' `cat system.txt`
PRLL_NR_CPUS="$JOBS" prll -s 'compile_single.sh "$1"' `cat world.txt` |
compile_single.sh
| Code: | #!/bin/bash
# Maximum system load to permit a new emerge process
MAX_LOAD="7"
LOAD=`cat /proc/loadavg | gawk '{ print $1; }'`
# BASH does not allow floating numbers to be compared, so use bc to compare system load
COND=`echo "$LOAD > $MAX_LOAD" | bc`
# if system load is to high we wait to other emerge process to finish
while [ "$COND" -eq "1" ]; do
sleep 5
LOAD=`cat /proc/loadavg | gawk '{ print $1; }'`
COND=`echo "$LOAD > $MAX_LOAD" | bc`
done
# Portage already have his log files, but if you need a new shiny one pick a location and a file name
LOG_FILE="/dev/null"
if (emerge --oneshot --nodeps ="$1" &>$LOG_FILE) ; then
printf "%-50s: %s\n" "$1" "OK" >> compile.log ;
else
printf "%-50s: %s\n" "$1" "FAIL" >> compile.log ;
fi |
A lot of things can goes wrong if your system is not up to date and dependencies had not yet taken into account.
But if every package need just to be recompiled it will work.
It's a crazy compile mode so you better have a backup.
It's working fine in my case but it's just one case. This method it's not very well tested (I mean a big users base who use it), so use it on your own risk.
The big problem is when A package depends by B package but A it's compiled before B, so be very carefully and better meet the prerequisites.
I'm not responsible if you will break your system 
Feel free to use the scripts, improve them, and, especially, let me know what you think about this method to recompile system and world sets.
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
 |
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Tue Aug 17, 2010 6:56 pm Post subject: |
|
|
| Got any statistics so we can compare? |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 402
|
| Posted: Tue Aug 17, 2010 7:36 pm Post subject: |
|
|
My statistics aren't very scientific and improvements depends a lot by everyone USE flags.
In my case system set contain 551 packages and 382 of thats are compiling once at a time.
emerge -e system with MAKEOPTS="-j8 --load-average=11 and EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=7.9 --keep-going --with-bdeps=y --complete-graph" it's done now in 1/3 of initial time.
And emerge -e world it's done in ~60% of time than previous.
Average system load never fall under 4.5
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
