Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[HOWTO] Hassle-free emerge -uD world (update-world v1.8)
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Sun Sep 10, 2006 11:51 pm    Post subject: [HOWTO] Hassle-free emerge -uD world (update-world v1.8) Reply with quote

New Version 1.8 (2/16/07)
I've added a lot of new features to the newest version of update-world. It seems pretty stable (to me). Let me know if there are any problems.
Changelog v1.8:
-Cleaned up and modularized code. It should be a lot more readable now.
-Script now sources /etc/make.conf. Several portage variables are used by this script now.
-Added ability to 'emerge -p <package> | update-world. (realize that this is really only meant for updating existing packages...if you're installing a new one, you probably can't just skip a package failure, as it's likely a required dependency)
-Added interface with portage 2.1's elog capability. You can type 'update-world --log' to view build messages after updating (mesages stored in ~/update-world/<date>/notice/. Check out http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=1#doc_chap4. if you haven't set up elog yet.
-Added interface with revdep-rebuild. You can run it alone (update-world --revdep) to prepare a list of packages to install from revdep rebuild, or you can use it in conjunction with --install (update-world --revdep --install) to automatically run revdep-rebuild when the update is complete.

About update-world
If your experience is anything like mine, the process of updating a Gentoo system is no small endeavour. Typing 'emerge -uD world' is an almost guaranteed day or two periodically checking on the process to make sure it hasn't died because of a compile failure. I recently updated my system (680 packages !!!), which took literally 3 days because I would find that after I started the process and went to bed/work/etc., it would die within a few hours of starting it. 10 packages wouldn't compile properly, causing me to have to restart the emerge process 10 times. This is obviously not very efficient.

This nightmarish update process led me to come up with something that makes things easier. "update-world" is a bash script that controls the update process and forces portage to skip past failed builds and compile until the update is finished. Failed builds are added to a list for review once the update is finished.
The script does depend on the 'equery' command, part of app-portage/gentoolkit.

Here's how to do it:
First, make the script executable and put it somewhere in your $PATH.
Then, run the script.
Code:
update-world --prepare
or
update-world --empty
or
emerge <package> | update-world

This begins the update process by creating a list of packages to be emerged, as produced by 'emerge -uD world'. (Alternatively, you can use 'update-world --empty' to create a list of all packages on the system, akin to 'emerge -e world'). This command creates a list called 'emergelist' in ~/.update-world/. Now, you can use your favorite editor to modify this list how you like--remove packages, change versions, etc. When you have the emergelist how you like it, you're ready to move onto the next step:
Code:
update-world --install

This command actually starts the build process, just as if you had typed emerge -uD world. The difference is, if a package fails for some reason or another, that package is added to a list 'failedlist' in ~/.update-world/. Also, a link to the portage log file for that package (usually located in /var/log/portage) is created in ~/.update-world/<date>/log/. Once the process is done, you'll be presented with a list of failed packages to take care of all at once, saving much time.

Now you can automatically run revdep-rebuild whenever you update with update-world. You can use it alone or in conjunction with --install.
Code:
update-world --revdep
or
update-world --revdep --install


Take advantage of portage 2.1's elog feature by running 'update-world --log' after an update to get those important messages that whiz by when doing a large update.
Look at http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=1#doc_chap4. for more information on setting this up. These notices are stored in ~/.update-world/notice.
Code:
update-world --log



If you ever need to stop the process, just hit 'Control-C' to abort the process. The update can then be restarted at any time, just where it left off, by retyping update-world --install.

I've only tested this on my machine, but it worked flawlessly, taking only 16 hours to compile all 680 packages. And the ten packages that wouldn't compile were waiting for me to fix at my leisure.

So, you can download the latest stable version of the script here: http://dixieflatline.homelinux.org/files/gentoo/update-world

Mirror 1: http://dixieflatline.homelinux.org/files/gentoo/
Mirror 2 (thanks ryker!): http://www.mindlesstechie.net/gentoo/update-world/

Featured in Gentoo Weekly Newsletter 12/04/06 8)
http://www.gentoo.org/news/en/gwn/20061204-newsletter.xml

Here's the code:
Code:

#!/bin/bash
# update-world 1.8
# count_zero 2007
# Licensed under the GPLv2
#
# update-world is a bash script that forces an emerge -uD world to continue, even
# in the face of (inevitable) compile failures.  Failed builds are added to a list,
# and the script jumps to the next package in the queue.
#
# just run update-world --prepare (or --empty) followed by update-world --install.
#
# Release Notes:
# v1.8: Cleaned and modularized code.  Added ability to utilize revdep-rebuild if wished.  Can now pipe other
#       emerges into update-world.  Sources /etc/make.conf.  Added interfacing with elog.  Some bugfixes.
# v1.7  broken.
# v1.6.1: Bugfixes, added utf-8 support
# v1.6: Script preserves colored output.  Fixed version numbering in help output.
# v1.5: now handles 'fetch restriction' properly
# v1.4: uses '--newuse' for emerges to respect new use flags
# v1.3: uses '--oneshot' for emerges so they don't get added to the world file.
# v1.2: Script now detects log directly instead of accepting only /var/log/portage
# v1.1: minor bugfix to allow package without version to be added to 'emergelist'
# v1.0: Release
#
#

#### Help ####
# Help information
displayhelp ()
{
echo "update-world 1.8
count_zero 2007
licensed under the GPLv2
http://www.fsf.org/licensing/licenses/gpl.txt

Description:
This is a bash script to automate the 'emerge -uD world' process.

The default program directory is ~/.update-world.
Error logs are filed in ~/.update-world/<date>/log.
Ebuild notices (elogs) are filed in ~/.update-world/<date>/notice.

Usage:
update-world [ --help | -h ]
   Display this help file and exit.
update-world [ --prepare | -p ]
   Prepares a list of packages produced by 'emerge -puvD world'
   and outputs them to the file 'emergelist' in the current directory.
   Add, remove, or change the version of any of the packages listed
   in this file with your favorite editor before continuing.
update-world [ --empty | -e ]
   Same as --prepare except it performs 'emerge -ev world' to recompile
   all packages on a system.
update-world [ --install | -i ]
   Begins the emerge process based upon the 'emergelist' file created
   from running 'update-world --prepare'.  Once finished, failed builds
   will be saved in the file 'failedlist' in the program directory.
update-world [ --revdep | -r ]
   Run revdep-rebuild to prepare the emergelist.  Can be used alone
   or in conjunction with --install.
emerge -p <package> | update-world
   Let portage produce an emergelist determined by a package.  This can be
   useful for large packages with lots of dependencies (kde, gnome, xorg, etc.).
   Once complete, run update-world --install.
update-world [ --log | -l ]
   View the ebuild notices (elogs) of packages installed by update-world,
   sorted by date.
"
}
#### end help ####

# Set the program directory and important variables
source /etc/make.conf
export EMERGE_DEFAULT_OPTS=""
homedir=~/.update-world
workdir="$homedir/`date +%F`"
logdir="$workdir/log"
noticedir="$workdir/notice"
mkdir -p $logdir
mkdir -p $noticedir
touch $homedir/failedlist
emergelist=$homedir/emergelist
emergetemp=$workdir/emergetemp
failedlist=$homedir/failedlist
parameter="-puvD --newuse"

#### error-check ####
# make sure there are no errors in the emerge process.
errorcheck ()
{
cat $emergelist | xargs emerge -p > $emergetemp
if [[ -n `cat $emergetemp | grep '\[blocks'` ]]
   then echo "WARNING: You have a blocking package.  Fix this before continuing."; exit 1
   exit 1
   elif grep -q "masked by:" $emergetemp
   then echo "WARNING: You may need to unmask some packages before continuing."; exit 1
   elif grep -q "emerge: there are no ebuilds to satisfy" $emergetemp
   then echo "WARNING: One of the specified packages doesn't exist"; exit 1
   elif [[ -n `cat $emergetemp | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*F[[:upper:][:lower:] ]*\].*/&/p'` ]]
   then fetchrestricted=`cat $emergetemp | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*F[[:upper:][:lower:] ]*\].*/&/p' | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p'`
   echo -e "\nWARNING: The following packages have Fetch Restriction turned on:\n$fetchrestricted"
   for each in $fetchrestricted
   do emerge =$each
   done
   exit 1
   else :
fi
}
#### end error-check ####

#### empty ####
# parameter for '--empty' to emerge -e world
empty ()
{
parameter='-pev'
}
#### end empty

#### prepare ####
# Prepare the list of packages to emerge into emerglist
prepare ()
{
if [[ -e "$emergelist" ]]
   then echo -en "The file 'emergelist' already exists and will be overwritten.\nContinue anyway? (Y/n) "
   read continue
   if [[ "$continue" = 'y' || "$continue" = 'yes' || "$continue" = 'Y' || "$continue" = 'Yes' || "$continue" = 'YES' || "$continue" = '' ]]
      then :
      else echo exiting.
      exit 0
   fi
   else :
fi
emerge $parameter world > $emergelist
errorcheck
emerge $parameter world
echo -e "\nThese are the packages that will be installed.  If you want to alter this list,\njust edit the file 'emergelist' in $homedir. When finished,\nrun 'update-world --install'"
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' | sort -d > $emergetemp
cp $emergetemp $emergelist
rm -f $failedlist
}
#### end prepare ####



#### installworld ####
# Install the packages listed in emergelist
installworld ()
{
# Make sure there's an emergelist
if [[ -s "$emergelist" ]]
   then :
   else echo "WARNING: no 'emergelist' exists.  Did you run 'update-world --prepare' first?"; exit 1
fi

errorcheck
condition=false
# set begin time
before_time="${before_time:=`date +%s`}"
# Until loop to make portage continue until finished
until [[ $condition = true ]]
do
   cat $emergelist | xargs emerge -1
   # Detect which packages have been successfully emerged and remove from the queue
   installed=`tac /var/log/emerge.log | sed '/Started\ emerge/,$d' | grep "completed emerge" | sed 's/^.*)\ //' | sed 's/\ to.*//'`
   for each in `cat "$emergelist" | sed 's/^=//'`
   do
      if [[ -n `echo "$installed" | grep "$each"` ]]
         then
         # Checking emerge logs for any ebuild notices
         einfo=""
         notice="$noticedir/${each/\//:}-notice.log"
         elog="${PORT_LOGDIR}/elog/`ls -lt $PORT_LOGDIR/elog | grep ${each/\//:} | sed '2,$ d' | sed 's/.*\ //'`"
         if [[ -n "$PORTAGE_ELOG_SYSTEM" ]] && [[ -n "${einfo:=`cat $elog | sed '/^$/d' | sed '/^[ ]*$/d' | sed 's/.*/ * &/'`}" ]]
            then
            echo ------------------------ > $notice
            echo -e "$each \n" >> $notice
            echo "$einfo" >> $notice
            echo ------------------------ >> $notice
            else :
         fi
         # Removing successful emerges
         each=`echo ${each//\//\\/} | sed 's/\-[0-9].*//'`
         cat $emergelist | sed "/${each//\//\\/}/d" > $emergetemp
         mv $emergetemp $emergelist
      fi
   done
   # See if all packages have been emerged.  If so, exit function "installworld"
   if [[ -z `cat "$emergelist"` ]]
      then condition=true
      break
      else :
   fi
   ## If there are remaining packages, find why the emerge stopped
   errorlog=`ls -t $PORT_LOGDIR | sed '2,$d'`

   # if user-initiated "control-c" just exit.
   if [[ -n `cat $PORT_LOGDIR/$errorlog | grep 'signal 2'` ]]
      then echo -e "\n*** User hit 'Control-C' ... exiting.\n"
      exit 1
      else :
   fi

   # if package failed, give user chance to stop update-world
   sleep 5
   echo
   echo "*** Hit Control-C to exit, or just wait to continue with emerge."
   echo
   sleep 10
   echo "*** Continuing with emerge."

   # Detect failed emerge and add to failedlist, remove failed package from emergelist
   failedpkg=`tac /var/log/emerge.log | sed '1d' | sed '/terminating/,$d' | sed '/completed\ emerge/,$d' | grep ">>>" | sed 's/.*)\ //' | sed 's/\ to.*//'`
   #### uh-oh, stop if something's really wrong to prevent infinite emerge (tried to emerge same package again)
   if [[ -n `tail -n 1 $failedlist | grep $failedpkg` ]]
      then echo "The failed package $failedpkg could not be merged and must be successfully installed before continuing."
      exit 1
      else :
   fi
   
   echo $failedpkg >> $failedlist
   # if package added to emergelist manually doesn't have a version, make sure it's still removed if it fails
   if [[ -z `cat "$emergelist" | grep "$failedpkg"` ]]
      then failedpkg2=`equery which =$failedpkg | sed 's/\/[^/]*$//' | sed -n 's/.*\/\([^/]*\/[^/]*\)/\1/p'`
      cat $emergelist | sed "/^${failedpkg2//\//\\/}/d" > $emergetemp
      else cat $emergelist | sed "/${failedpkg//\//\\/}/d" > $emergetemp
   fi
   mv $emergetemp $emergelist
   # link to the log file of the failed package
   ln -s $PORT_LOGDIR/$errorlog $logdir/${errorlog}_error-log
   echo
   echo
   echo "*** $failedpkg compile failed, skipping."
   # Is there a package in 'emergelist' that REQUIRES the failed package as a dep? If so, remove it too.
   failedpkgdeps=`equery which =$failedpkg | sed 's/\/[^/]*$//' | sed -n 's/.*\/\([^/]*\/[^/]*\)/\1/p'`
   deps=`equery depends $failedpkgdeps | sed '/^\[/d' | sed 's/.*/=&/'`

   for each in `echo $deps`
   do
      if [[ -n `cat $emergelist | grep "$each"` ]]
         then
         if [[ -n `emerge -p $each | grep "$failedpkg"` ]]
            then
            each2=`cat $emergelist | grep $each | sed 's/\=//'`
            echo "$each2 (depends on $failedpkg)" >> $failedlist
            cat $emergelist | sed "/${each//\//\\/}/d" > $emergetemp
            mv $emergetemp $emergelist
            echo "*** $each depends on $failedpkg, skipping."
            else :
         fi
         else :
      fi
   done
   if [[ -z `cat "$emergelist"` ]]
      then condition=true
      break
      else :
   fi
   echo "*** Continuing with emerge world"
done
}
#### end installworld ####

#### revdep-rebuild ####
# this script can call revdep-rebuild and get it's emergelist from there.
# useful to run in conjunction with update-world --install, it will automatically
# run revdep-rebuild after the emerge -uD world is complete
revdep ()
{
rm /root/.revdep-rebuild*.?_*
revdep-rebuild --package-names --pretend | tee $emergelist
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' > $emergetemp
mv $emergetemp $emergelist
}
#### end revdep-rebuild ####

#### piped ####
# emergelist is piped from another emerge (i.e., emerge kde | update-world)
piped ()
{
echo "$piped" > $emergelist
errorcheck
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' > $emergetemp
mv $emergetemp $emergelist
cat $emergelist | xargs emerge -p
echo -e "\nThese are the packages that will be installed.  If you want to alter this list,\njust edit the file 'emergelist' in $homedir. When finished, run\n'update-world --install'"
}
#### end piped ####

#### finished ####
# Display closing messages
finished ()
{
echo -e "\n\nCongratulations!  'update-world' complete."
elapsed_time
echo
# All done! display the failed packages, if any
if [[ -z `cat $failedlist` ]]
   then echo "All packages emerged successfully"
   else echo "These packages couldn't be merged due to compile errors:"
   echo
   cat $failedlist
   echo
   echo -e "Look in $logdir/ for the portage log files of the failed builds.\nCheck bugs.gentoo.org or the Gentoo Forums for help."
fi
if [[ -n "$PORTAGE_ELOG_SYSTEM" ]]
   then echo -e "\n * To review portage elog notices, type \"update-world --log\"\n"
   else echo " * To enable portage notice viewing with elog, check out http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=1#doc_chap4."
fi
}
#### end finished ####

#### elapsed_time ####
# just for fun let's see how long the process takes...
elapsed_time ()
{
after_time=`date +%s`

time=$(( after_time - before_time ))

days=$(( time / 86400 ))
time=$(( time - ( days * 86400 ) ))
hours=$(( time / 3600 ))
time=$(( time - ( hours * 3600 ) ))
minutes=$(( time / 60 ))
time=$(( time - ( minutes * 60 ) ))
seconds=$time

if [[ "$days" -eq 0 ]]
   then days=""
elif [[ "$days" -eq 1 ]]
   then days="$days day, "
elif [[ "$days" -gt 1 ]]
   then days="$days days, "
fi

if [[ "$hours" -eq 0 ]]
   then hours=""
elif [[ "$hours" -eq 1 ]]
   then hours="$hours hour, "
elif [[ "$hours" -gt 1 ]]
   then hours="$hours hours, "
fi

if [[ "$minutes" -eq 0 ]]
   then minutes=""
elif [[ "$minutes" -eq 1 ]]
   then minutes="$minutes minute, and "
elif [[ "$minutes" -gt 1 ]]
   then minutes="$minutes minutes, and "
fi

seconds="$seconds seconds"

echo -e "Total compiling time: $days$hours$minutes$seconds.\n"
}
#### end elapsed_time ####

#### What should the script do when invoked? ####
# I have nine functions: displayhelp, empty, prepare, errorcheck, installworld, revdep, piped, elapsed_time, and finished

arguments="$@"

# update-world --help
if [[ -n `echo "$arguments" | sed -n 's/-h/&/p'` || -n `echo "$arguments" | sed -n 's/--help/&/p'` ]]
   then
   displayhelp
   exit 0
   else :
fi

# update-world --empty
if [[ -n `echo -e \\t "$arguments" | sed -n 's/-e/&/p'` || -n `echo "$arguments" | sed -n 's/--empty/&/p'` ]]
   then echo Preparing list of packages...
   empty
   prepare
   exit 0
   else :
fi

# update-world --prepare
if [[ -n `echo "$arguments" | sed -n 's/-p/&/p'` || -n `echo "$arguments" | sed -n 's/--prepare/&/p'` ]]
   then echo Preparing list of packages...
   prepare
   exit 0
   else :
fi

# update-world --install
if [[ -n `echo "$arguments" | sed -n 's/-[[:lower:]]*i/&/p'` || -n `echo "$arguments" | sed -n 's/--install/&/p'` ]]
   then
   installworld
   check=true
   else :
fi

# update-world --revdep
if [[ -n `echo "$arguments" | sed -n 's/-[[:lower:]]*r/&/p'` || -n `echo "$arguments" | sed -n 's/--revdep/&/p'` ]]
   then
   revdep
   if [[ -s "$emergelist" ]]
      then installworld
      else :
   fi
   check=true
   else :
fi

# emerge -p <package> | update-world
# piped command MUST include --pretend !!!
if [[ "$arguments" = "" ]]
   then echo "No arguments supplied.  If you're piping from the emerge command (emerge -p <package> | update-world), this is fine.  Otherwise, cancel and type 'update-world --help' for usage information."
   until [[ `echo "$piped" | wc -l` -eq 10 ]] && [[ -z `echo "$piped" | grep 'Calculating dependencies'` ]] || [[ -z `echo "${piped:-1}" | tail -n 4` ]]
   do read line
   piped="${piped}
$line"
   done
   else :
fi

if [[ -n `echo "$piped" | sed -n 's/Calculating\ dependencies/&/p'` ]]
   then
   piped
   exit 0
   else :
fi

# update-world --log
if [[ -n `echo "$arguments" | sed -n 's/-l/&/p'` || -n `echo "$arguments" | sed -n 's/--log/&/p'` ]]
then echo "Choose a date: "
select directory in `ls -Al | grep ^d | sed 's/.*\ //'`
do
   if [[ -n "$directory" ]]
      then date=$directory
      break
      else echo "invalid selection"
   fi
done
rm -f $homedir/$date/notice/log
for each in `ls $homedir/$date/notice/`
do cat $homedir/$date/notice/$each >> $homedir/$date/notice/log
done
less $homedir/$date/notice/log
check=true
exit 0
else :
fi

# Check if arguments were valid
if [[ ${check:-false} == false ]]
   then echo -e "Error: Invalid Command.\nRun 'update-world --help' for usage."
   exit 1
   else :
fi

# done! Display closing text
finished
exit 0


Let me know if you find any bugs or have feedback. Good luck! :D
[Update] 9/13/06 (version 1.1)
Fixed script to prevent errors when adding a package without specifying a version number to 'emergelist'.
[Update] 10/27/06 (version 1.2)
Script now automatically detects the portage log directory instead of just assuming /var/log/portage.
[Update] 11/22/06 (version 1.4) I guess my edit didn't take yesterday for v1.3... 8O
Added --newuse and --oneshot to emerge variables.
[Update] 11/22/06 (version 1.5)
Now handles 'fetch restriction' properly, prompting user to download sources before continuing.
[Update] 12/06/06 (version 1.6)
Script preserves colored emerge output. Fixed version numbering in help output.
[Update] 12/09/06 (version 1.7)
Cleaned up the code a little, fixed some bugs. Check out the new feature for managing USE flags!
Edit: seems to be broken :(
[Update] 12/10/06 (version 1.6.1)
bugfixes and utf-8 support.
[Update] 2/16/07 (version 1.8)
Cleaned and modularized code. Added ability to utilize revdep-rebuild if wished. Can now pipe other
emerges into update-world. Sources /etc/make.conf. Added interfacing with elog. Some bugfixes.
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin


Last edited by count_zero on Fri Feb 16, 2007 1:38 pm; edited 16 times in total
Back to top
View user's profile Send private message
therealkurisu
n00b
n00b


Joined: 11 Sep 2006
Posts: 2
Location: Mesa, AZ

PostPosted: Mon Sep 11, 2006 1:51 pm    Post subject: A wonderful solution to a common problem! Reply with quote

I'm fairly new to Gentoo (and linux in general ^_^) but I find this script to be VERY useful. I've ran it 5 or so times so far and haven't had any problems at all. It saves SO much time from compiling errors and general problems (and allows for skipping updates on known problematic areas!).

Automated skipping, ftw. I reccomend that everyone try it out.

Good job count_zero.


(Also, <3 Crono and therefore, through association, you :P)
Back to top
View user's profile Send private message
Doogman
Apprentice
Apprentice


Joined: 24 Sep 2004
Posts: 244

PostPosted: Mon Sep 11, 2006 8:05 pm    Post subject: Reply with quote

This looks like a good idea.

While my dual-core system can compile packages at a good clip, rebuilding world always stalls out at some point requiring user intervention. Instead of grinding away at the build list while I'm at work, the computer spends a good deal of time at idle until I get home again. With this script, most of the work should be done while I'm away.

Maybe I'll finally tackle that gcc upgrade I've been putting off.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 3003
Location: Bay Area, CA

PostPosted: Tue Sep 12, 2006 8:47 pm    Post subject: Reply with quote

Removing deps of the failed pkgs is definitely better idea than portage's 'emerge --resume --skipfirst' which can break your system in no time.
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Sun Sep 17, 2006 2:21 am    Post subject: Reply with quote

It was discussed in another thread about combining this script with emerging packages in the correct order, a la this thread by Guenther Brunthaler (thanks for a job well done, Guenther :D).

If you run Guenther's script up until the point at which you will start rebuilding your system (gcc already emerged), you'll end up with a file '/root/recompile-remaining-packages'. This file contains all of the packages to be emerged, in the order to best rebuild the toolchain and the rest of the system. It's pretty easy to interface this with update-world:
Code:
grep "item " /root/recompile-remaining-packages | sed 's/item\ .*\ //' > ~/.update-world/emergelist

and then
Code:
update-world --install


Have fun with interruption-free emerging of your system in the "correct" order. 8)
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
Bynds
n00b
n00b


Joined: 29 Sep 2006
Posts: 1

PostPosted: Fri Sep 29, 2006 12:53 pm    Post subject: My Aspirin Reply with quote

This script has just cured every headache I had with my Gentoo box (I'm not a heavy user :D )
It took me weeks to update my old compaq server when I dusted it off after about 2 years in the cupboard, this would have saved me about half my time.

Fantastic!
Back to top
View user's profile Send private message
bender02
Tux's lil' helper
Tux's lil' helper


Joined: 05 May 2006
Posts: 82
Location: Nashville, TN, USA

PostPosted: Thu Oct 26, 2006 10:19 pm    Post subject: Reply with quote

Great script, thanks!

I have one thing though, while running "update-world --install", I get an error like
Code:
cat: /var/log/portage/elog: Is a directory

and the dir inside .update-world which is supposed to contain the logs for failed packages contains only a link to /var/log/portage/elog.
My logs structure for elog is that /var/log/portage/elog is a directory containing logs for various emerges. Is there an easy way to fix this? Thanks.
_________________
(I love deadlines. I like the whooshing sound they make as they fly by. [DA])
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Fri Oct 27, 2006 8:03 pm    Post subject: Reply with quote

The problem was that the script assumed that /var/log/portage was the portage log directory. I've updated the script so that it detects the setting of PORT_LOGDIR in /etc/make.conf. I've updated the code in the first post, but my (free) webhost is giving me some problems so I can't update the download yet :(
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
bender02
Tux's lil' helper
Tux's lil' helper


Joined: 05 May 2006
Posts: 82
Location: Nashville, TN, USA

PostPosted: Sat Oct 28, 2006 3:57 am    Post subject: Reply with quote

Great! Thanks! (But I'm gonna get to test it only next time I sync...)
_________________
(I love deadlines. I like the whooshing sound they make as they fly by. [DA])
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Sat Oct 28, 2006 5:42 pm    Post subject: Reply with quote

Download link fixed!
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
ryker
Guru
Guru


Joined: 28 May 2003
Posts: 412
Location: Portage, IN

PostPosted: Sat Nov 11, 2006 3:56 pm    Post subject: Reply with quote

This script needs more exposure! This is the most useful emerge wrapper that I have seen yet. They should feature it on GWN sometime.

Thank you for your time and effort on this.
_________________
Athlon 64 3200+, 80G WD sata hd + 200G IDE, 1G Geil DDR400, MSI K8T Neo
IntelCore2Duo 2.0Ghz MSI laptop,100G SATA hd, 2G RAM
Back to top
View user's profile Send private message
richfish
Apprentice
Apprentice


Joined: 03 Mar 2006
Posts: 202
Location: Phoenix, AZ

PostPosted: Mon Nov 20, 2006 8:59 pm    Post subject: Reply with quote

Shouldn't this script use --oneshot for merging packages so it doesn't pollute the world file with all the packages it updates?
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Tue Nov 21, 2006 3:19 am    Post subject: Reply with quote

Good idea!

Easy to implement too. I just added '-1' to the end of line 101 in the script.
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
RageOfOrder
Tux's lil' helper
Tux's lil' helper


Joined: 06 Aug 2006
Posts: 99
Location: EH?!?!

PostPosted: Tue Nov 21, 2006 9:29 am    Post subject: Reply with quote

Wow, this is just awesome. I've been putting this off for weeks. I think I'll finally update everything on wednesday night.

Thanks for all your hard work!
_________________
...And then stuff happened.
Back to top
View user's profile Send private message
dkure
n00b
n00b


Joined: 20 Apr 2004
Posts: 51
Location: Sydney, Australia

PostPosted: Wed Nov 22, 2006 8:07 am    Post subject: Reply with quote

It would be great if there was two more features.

This script should include --newuse, to make sure use flags are respected.

Second is more of a request, and is a package fetch option. This would fetch the packages selected in the .update_world file. Allowing packages to be downloaded in the background overnight before packages are selected if need be.

I agree that this script should certainly get featured on GNN, makes updating simpler and more specialized to peoples needs.
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Wed Nov 22, 2006 2:54 pm    Post subject: Reply with quote

Quote:
This script should include --newuse, to make sure use flags are respected.

I guess the meaning of "--newuse" has changed since I last read the man page...I thought that (when used in conjunction with --update world) it updated ONLY those packages that would be affected by recently changed USE flags.
From the Gentoo Wiki (http://gentoo-wiki.com/MAN_emerge):
Quote:
--newuse (-N)
Tells emerge to include installed packages where USE flags have changed since compilation. An asterisk marks when a USE flag has changed since the package was compiled.


However, the gentoo handbook suggests that it might be useful in better updating an entire system.
From the installation handbook (http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1):
Quote:
Code Listing 14: Updating your entire system

# emerge --update --deep world


Since security updates also happen in packages you have not explicitly installed on your system (but that are pulled in as dependencies of other programs), it is recommended to run this command once in a while.

If you have altered any of your USE flags lately you might want to add --newuse as well. Portage will then verify if the change requires the installation of new packages or recompilation of existing ones:


So i guess I'm a little confused about whether --newuse would be helpful in this situation, where all packages will be updated anyway. Doesn't portage look a new use flags anyway? Maybe someone could clarify this? Implementing this in the script is pretty trivial.

Quote:
Second is more of a request, and is a package fetch option. This would fetch the packages selected in the .update_world file. Allowing packages to be downloaded in the background overnight before packages are selected if need be.

I'll look into this, parallel fetching would be a useful feature. I believe it is going to be built into portage sometime in the future, but it could probably be implemented here. I'll let you know.

Quote:
I agree that this script should certainly get featured on GNN, makes updating simpler and more specialized to peoples needs.

Thanks for all the helpful feedback everyone, it makes doing stuff like this very enjoyable. :D
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
ryker
Guru
Guru


Joined: 28 May 2003
Posts: 412
Location: Portage, IN

PostPosted: Wed Nov 22, 2006 4:05 pm    Post subject: Reply with quote

I believe --newuse also implies --update. Of course, you can always add additional parameters such as -D to catch more packages. I added --newuse to the update-world script on my system and it works fine.

I just changed
Code:

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD'


to
Code:

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD --newuse'


I love this script so much, I was planning on writing and submitting a tip to the GWN this weekend.
_________________
Athlon 64 3200+, 80G WD sata hd + 200G IDE, 1G Geil DDR400, MSI K8T Neo
IntelCore2Duo 2.0Ghz MSI laptop,100G SATA hd, 2G RAM
Back to top
View user's profile Send private message
padoor
Advocate
Advocate


Joined: 30 Dec 2005
Posts: 4185
Location: india

PostPosted: Wed Nov 22, 2006 4:53 pm    Post subject: Reply with quote

this is looking good for me too.
please instruct how do i install and run this .
which script do i download and how i can use it?
i need codes for full proceedure.
i am still newbee for these operations.
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)
Back to top
View user's profile Send private message
ryker
Guru
Guru


Joined: 28 May 2003
Posts: 412
Location: Portage, IN

PostPosted: Wed Nov 22, 2006 8:55 pm    Post subject: Reply with quote

It's pretty straight forward and the instructions are clearly given in the first post.

Basically, all you have to do is:

    1. Download the latest version of the script from here. and save it as /usr/bin/update-world.
    2. chmod +x /usr/bin/update-world


That's it. Now just type update-world and it tells you what to do.
_________________
Athlon 64 3200+, 80G WD sata hd + 200G IDE, 1G Geil DDR400, MSI K8T Neo
IntelCore2Duo 2.0Ghz MSI laptop,100G SATA hd, 2G RAM
Back to top
View user's profile Send private message
padoor
Advocate
Advocate


Joined: 30 Dec 2005
Posts: 4185
Location: india

PostPosted: Thu Nov 23, 2006 1:22 am    Post subject: Reply with quote

that was pretty easy
and it works noww.
i am due for world update in 2 days.
i will post the result , thanks for direction ryker :)
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)
Back to top
View user's profile Send private message
padoor
Advocate
Advocate


Joined: 30 Dec 2005
Posts: 4185
Location: india

PostPosted: Thu Nov 23, 2006 2:28 am    Post subject: Reply with quote

i did an emerge --sync
then emerge -Du systaem.
update-world -p made a file list as follows
localhost ramaswamy # update-world -p
The file 'emergelist' already exists and will be overwritten.
Continue anyway? (y/n) y

These are the packages that would be merged, in order:

Calculating world dependencies ..... ..... ..
!!! Ebuilds for the following packages are either all
!!! masked or don't exist:
net-im/ymessenger

... done!
[ebuild U ] dev-java/sun-jdk-1.5.0.09-r1 [1.5.0.08] USE="X alsa -doc -examples -jce -nsplugin" 32,469 kB
[ebuild N ] virtual/jdk-1.5.0 0 kB
[ebuild U ] x11-themes/gnome-backgrounds-2.16.2 [2.16.1] USE="-debug" 1,172 kB
[ebuild U ] dev-java/sun-jre-bin-1.5.0.09-r1 [1.5.0.08-r1] USE="X alsa -nsplugin" 0 kB
[ebuild U ] dev-cpp/gtkmm-2.10.2-r1 [2.8.3] USE="-accessibility% -debug -doc% -examples%" 0 kB
[ebuild FU ] dev-java/sun-jdk-1.4.2.13 [1.4.2.12-r2] USE="X alsa -doc -examples -jce -nsplugin" 35,509 kB
[ebuild U ] www-client/mozilla-firefox-bin-2.0 [1.5.0.7] LINGUAS="-ar -bg% -ca -cs -da -de -el -en_GB -es -es_AR -es_ES -eu% -fi -fr -fy% -fy_NL% -ga -ga_IE -gu% -gu_IN% -hu -it -ja -ko -lt% -mk -mn% -nb -nb_NO -nl -nn% -nn_NO% -pl -pt% -pt_BR -pt_PT% -ru -sk -sl -sv -sv_SE -tr -zh% -zh_CN -zh_TW (-he%) (-ro%)" 0 kB
[ebuild U ] gnome-base/gnome-vfs-2.16.3 [2.16.2] USE="hal ipv6 ssl -avahi -debug -doc -gnutls -samba" 1,855 kB
[ebuild U ] gnome-base/gnome-desktop-2.16.2 [2.16.1] USE="-debug -doc" 1,202 kB
[ebuild U ] gnome-base/eel-2.16.3 [2.16.1] USE="X -debug" 657 kB
[ebuild U ] gnome-base/nautilus-2.16.3 [2.16.2-r1] USE="X gnome -beagle -debug" 4,168 kB
[ebuild U ] gnome-extra/evolution-data-server-1.8.2 [1.8.1-r1] USE="ipv6 ldap ssl -debug -doc -kerberos -keyring -krb4 -nntp" 6,800 kB
[ebuild U ] gnome-extra/nautilus-cd-burner-2.16.2 [2.16.1-r1] USE="cdr dvdr -debug" 698 kB
[ebuild U ] media-libs/xine-lib-1.1.3_pre20061119 [1.1.3_pre20061112] USE="X aac aalib alsa arts dvd esd gnome gtk ipv6 mad nls opengl oss sdl truetype vorbis win32codecs xv -a52 (-altivec) -debug -directfb -dts -dxr3 -fbcon -flac -imagemagick -libcaca -mmap -mng -modplug -pulseaudio -samba -speex -theora -v4l -vcd -vidix -xinerama -xvmc" 6,814 kB
[ebuild U ] gnome-base/control-center-2.16.2 [2.16.1] USE="alsa eds -debug" 1,975 kB
[ebuild U ] gnome-extra/gnome-games-2.16.2 [2.16.1.1] USE="-artworkextra -debug -guile (-avahi%)" 5,134 kB
[ebuild U ] www-client/epiphany-2.16.2 [2.16.1] USE="python -debug -doc" 4,130 kB
[ebuild U ] media-sound/sound-juicer-2.16.2 [2.16.1] USE="ogg -debug -flac" 1,236 kB
[ebuild U ] app-arch/file-roller-2.16.2 [2.16.1] USE="gnome%* -debug" 1,105 kB
[ebuild U ] gnome-extra/gtkhtml-3.12.2 [3.12.1] USE="-debug -static" 1,255 kB
[ebuild U ] mail-client/evolution-2.8.2.1 [2.8.1.1] USE="crypt dbus hal ipv6 ldap spell ssl -bogofilter -debug -doc -kerberos -krb4 -mono -nntp -pda -profile" 12,618 kB
[ebuild U ] net-misc/knemo-0.4.6 [0.4.5] USE="arts -debug -wifi -xinerama" 589 kB
[ebuild U ] kde-base/nsplugins-3.5.5-r1 [3.5.5] USE="arts -debug -kdeenablefinal -kdehiddenvisibility -xinerama" 0 kB

Total: 23 packages (22 upgrades, 1 new), Size of downloads: 119,393 kB
Fetch Restriction: 1 package (1 unsatisfied)

These are the packages that will be installed. If you want to alter this list,
just edit the file 'emergelist' in /root/.update-world. When finished, run
'update-world --install'
localhost ramaswamy #

localhost ramaswamy # update-world --install
Calculating dependencies
emerge: there are no ebuilds to satisfy "=Fetch".

cat: /wget-log.3: No such file or directory

*** Hit Control-C to exit, or just wait to continue with emerge.

*** Continuing with emerge.
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
sed: -e expression #1, char 0: no previous regular expression
ln: creating symbolic link `/root/.update-world/2006-11-23/log/wget-log.3_error-log': File exists


*** compile failed, skipping.
grep: invalid option -- ,
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unrecognized option `--all-packages'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown directories method
grep: option `--direct' requires an argument
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown devices method
grep: unrecognized option `--indirect'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
*** Continuing with emerge world


Usage:
emerge [ options ] [ action ] [ ebuildfile | tbz2file | dependency ] [ ... ]
emerge [ options ] [ action ] < system | world >
emerge < --sync | --metadata | --info >
emerge --resume [ --pretend | --ask | --skipfirst ]
emerge --help [ system | world | config | --sync ]
Options: -[abBcCdDefgGhikKlnNoOpqPsStuvV] [--oneshot] [--newuse] [--noconfmem]
[ --color < y | n > ] [ --columns ]
[--nospinner]
[ --deep ] [--with-bdeps < y | n > ]
Actions: [ --clean | --depclean | --prune | --regen | --search | --unmerge ]


For more help try 'emerge --help' or consult the man page.

Congratulations! 'emerge world' complete.

All packages emerged successfully
localhost ramaswamy #

is this my options error or the fetch has some problem?
regular emerge -Du world does fetch the files and started with jdk

what mistake i am doing??
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Thu Nov 23, 2006 4:33 am    Post subject: Reply with quote

ryker wrote:
I believe --newuse also implies --update. Of course, you can always add additional parameters such as -D to catch more packages. I added --newuse to the update-world script on my system and it works fine.

I just changed
Code:

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD'


to
Code:

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD --newuse'



Done!
I tested it also, and seems to work well.
Version 1.4 now reflects these changes. The script above and the download link are updated.
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Thu Nov 23, 2006 4:48 am    Post subject: Reply with quote

@padoor:

you've uncovered a couple of problems, I think they are pretty easily fixed.
Quote:
Calculating world dependencies ..... ..... ..
!!! Ebuilds for the following packages are either all
!!! masked or don't exist:
net-im/ymessenger



Portage isn't finding this package for your system. It's probably masked on your system, and *could* (though maybe it shouldn't, find out why it's masked) be unmasked by adding 'net-im/ymessenger' to /etc/portage/package.unmask.

Quote:

[ebuild FU ] dev-java/sun-jdk-1.4.2.13 [1.4.2.12-r2] USE="X alsa -doc -examples -jce -nsplugin" 35,509 kB
. . .
Fetch Restriction: 1 package (1 unsatisfied)


You'll need to download the sources for sun-jdk from the website and copy to /usr/portage/distfiles/ before running the script.
Fix these problems before re-running update-world.

I should be able to make some changes to the script so it handles these errors better.
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
padoor
Advocate
Advocate


Joined: 30 Dec 2005
Posts: 4185
Location: india

PostPosted: Thu Nov 23, 2006 5:07 am    Post subject: Reply with quote

i will finish emerge of jdk and stop the emerge -Du world and try again.
ymessenger is full masked and i have the latest version installed. so it can skip this file
lets see after i finish this jdk emerge,
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)
Back to top
View user's profile Send private message
count_zero
Guru
Guru


Joined: 17 May 2004
Posts: 460
Location: Little Rock, Arkansas, USA

PostPosted: Thu Nov 23, 2006 6:22 am    Post subject: Reply with quote

The script version 1.5 (second one tonight 8O) properly handles "Fetch Restriction" now, prompting you to download the sources before continuing.
_________________
"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3, 4, 5, 6  Next
Page 1 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum