[HOWTO] Hassle-free emerge -uD world (update-world v1.8)

[count_zero]

New Version 1.8 (2/16/07)
I've added a lot of new features to the newest version of update-world. It seems pretty stable (to me). Let me know if there are any problems.
Changelog v1.8:
-Cleaned up and modularized code. It should be a lot more readable now.
-Script now sources /etc/make.conf. Several portage variables are used by this script now.
-Added ability to 'emerge -p <package> | update-world. (realize that this is really only meant for updating existing packages...if you're installing a new one, you probably can't just skip a package failure, as it's likely a required dependency)
-Added interface with portage 2.1's elog capability. You can type 'update-world --log' to view build messages after updating (mesages stored in ~/update-world/<date>/notice/. Check out http://www.gentoo.org/doc/en/handbook/h ... doc_chap4. if you haven't set up elog yet.
-Added interface with revdep-rebuild. You can run it alone (update-world --revdep) to prepare a list of packages to install from revdep rebuild, or you can use it in conjunction with --install (update-world --revdep --install) to automatically run revdep-rebuild when the update is complete.

About update-world
If your experience is anything like mine, the process of updating a Gentoo system is no small endeavour. Typing 'emerge -uD world' is an almost guaranteed day or two periodically checking on the process to make sure it hasn't died because of a compile failure. I recently updated my system (680 packages !!!), which took literally 3 days because I would find that after I started the process and went to bed/work/etc., it would die within a few hours of starting it. 10 packages wouldn't compile properly, causing me to have to restart the emerge process 10 times. This is obviously not very efficient.

This nightmarish update process led me to come up with something that makes things easier. "update-world" is a bash script that controls the update process and forces portage to skip past failed builds and compile until the update is finished. Failed builds are added to a list for review once the update is finished.
The script does depend on the 'equery' command, part of app-portage/gentoolkit.

Here's how to do it:
First, make the script executable and put it somewhere in your $PATH.
Then, run the script.

Code: Select all

update-world --prepare
or
update-world --empty
or
emerge <package> | update-world

This begins the update process by creating a list of packages to be emerged, as produced by 'emerge -uD world'. (Alternatively, you can use 'update-world --empty' to create a list of all packages on the system, akin to 'emerge -e world'). This command creates a list called 'emergelist' in ~/.update-world/. Now, you can use your favorite editor to modify this list how you like--remove packages, change versions, etc. When you have the emergelist how you like it, you're ready to move onto the next step:

Code: Select all

update-world --install

This command actually starts the build process, just as if you had typed emerge -uD world. The difference is, if a package fails for some reason or another, that package is added to a list 'failedlist' in ~/.update-world/. Also, a link to the portage log file for that package (usually located in /var/log/portage) is created in ~/.update-world/<date>/log/. Once the process is done, you'll be presented with a list of failed packages to take care of all at once, saving much time.

Now you can automatically run revdep-rebuild whenever you update with update-world. You can use it alone or in conjunction with --install.

Code: Select all

update-world --revdep
or
update-world --revdep --install

Take advantage of portage 2.1's elog feature by running 'update-world --log' after an update to get those important messages that whiz by when doing a large update.
Look at http://www.gentoo.org/doc/en/handbook/h ... doc_chap4. for more information on setting this up. These notices are stored in ~/.update-world/notice.

Code: Select all

update-world --log

If you ever need to stop the process, just hit 'Control-C' to abort the process. The update can then be restarted at any time, just where it left off, by retyping update-world --install.

I've only tested this on my machine, but it worked flawlessly, taking only 16 hours to compile all 680 packages. And the ten packages that wouldn't compile were waiting for me to fix at my leisure.

So, you can download the latest stable version of the script here: http://dixieflatline.homelinux.org/file ... date-world

Mirror 1: http://dixieflatline.homelinux.org/files/gentoo/
Mirror 2 (thanks ryker!): http://www.mindlesstechie.net/gentoo/update-world/

Featured in Gentoo Weekly Newsletter 12/04/06
http://www.gentoo.org/news/en/gwn/20061 ... letter.xml

Here's the code:

Code: Select all

#!/bin/bash
# update-world 1.8
# count_zero 2007
# Licensed under the GPLv2
#
# update-world is a bash script that forces an emerge -uD world to continue, even
# in the face of (inevitable) compile failures.  Failed builds are added to a list,
# and the script jumps to the next package in the queue.
#
# just run update-world --prepare (or --empty) followed by update-world --install.
#
# Release Notes:
# v1.8: Cleaned and modularized code.  Added ability to utilize revdep-rebuild if wished.  Can now pipe other
#       emerges into update-world.  Sources /etc/make.conf.  Added interfacing with elog.  Some bugfixes.
# v1.7  broken.
# v1.6.1: Bugfixes, added utf-8 support
# v1.6: Script preserves colored output.  Fixed version numbering in help output.
# v1.5: now handles 'fetch restriction' properly
# v1.4: uses '--newuse' for emerges to respect new use flags
# v1.3: uses '--oneshot' for emerges so they don't get added to the world file.
# v1.2: Script now detects log directly instead of accepting only /var/log/portage
# v1.1: minor bugfix to allow package without version to be added to 'emergelist'
# v1.0: Release
#
#

#### Help ####
# Help information
displayhelp ()
{
echo "update-world 1.8
count_zero 2007
licensed under the GPLv2
http://www.fsf.org/licensing/licenses/gpl.txt

Description:
This is a bash script to automate the 'emerge -uD world' process.

The default program directory is ~/.update-world.
Error logs are filed in ~/.update-world/<date>/log.
Ebuild notices (elogs) are filed in ~/.update-world/<date>/notice.

Usage:
update-world [ --help | -h ]
	Display this help file and exit.
update-world [ --prepare | -p ]
	Prepares a list of packages produced by 'emerge -puvD world'
	and outputs them to the file 'emergelist' in the current directory.
	Add, remove, or change the version of any of the packages listed
	in this file with your favorite editor before continuing.
update-world [ --empty | -e ]
	Same as --prepare except it performs 'emerge -ev world' to recompile
	all packages on a system.
update-world [ --install | -i ]
	Begins the emerge process based upon the 'emergelist' file created 
	from running 'update-world --prepare'.  Once finished, failed builds
	will be saved in the file 'failedlist' in the program directory.
update-world [ --revdep | -r ]
	Run revdep-rebuild to prepare the emergelist.  Can be used alone
	or in conjunction with --install.
emerge -p <package> | update-world
	Let portage produce an emergelist determined by a package.  This can be
	useful for large packages with lots of dependencies (kde, gnome, xorg, etc.).
	Once complete, run update-world --install.
update-world [ --log | -l ]
	View the ebuild notices (elogs) of packages installed by update-world,
	sorted by date.
"
}
#### end help ####

# Set the program directory and important variables
source /etc/make.conf
export EMERGE_DEFAULT_OPTS=""
homedir=~/.update-world
workdir="$homedir/`date +%F`"
logdir="$workdir/log"
noticedir="$workdir/notice"
mkdir -p $logdir
mkdir -p $noticedir
touch $homedir/failedlist
emergelist=$homedir/emergelist
emergetemp=$workdir/emergetemp
failedlist=$homedir/failedlist
parameter="-puvD --newuse"

#### error-check ####
# make sure there are no errors in the emerge process.
errorcheck ()
{
cat $emergelist | xargs emerge -p > $emergetemp
if [[ -n `cat $emergetemp | grep '\[blocks'` ]]
	then echo "WARNING: You have a blocking package.  Fix this before continuing."; exit 1
	exit 1
	elif grep -q "masked by:" $emergetemp
	then echo "WARNING: You may need to unmask some packages before continuing."; exit 1
	elif grep -q "emerge: there are no ebuilds to satisfy" $emergetemp
	then echo "WARNING: One of the specified packages doesn't exist"; exit 1
	elif [[ -n `cat $emergetemp | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*F[[:upper:][:lower:] ]*\].*/&/p'` ]]
	then fetchrestricted=`cat $emergetemp | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*F[[:upper:][:lower:] ]*\].*/&/p' | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p'`
	echo -e "\nWARNING: The following packages have Fetch Restriction turned on:\n$fetchrestricted"
	for each in $fetchrestricted
	do emerge =$each
	done
	exit 1
	else :
fi
}
#### end error-check ####

#### empty ####
# parameter for '--empty' to emerge -e world
empty ()
{
parameter='-pev'
}
#### end empty

#### prepare ####
# Prepare the list of packages to emerge into emerglist
prepare ()
{
if [[ -e "$emergelist" ]]
	then echo -en "The file 'emergelist' already exists and will be overwritten.\nContinue anyway? (Y/n) "
	read continue
	if [[ "$continue" = 'y' || "$continue" = 'yes' || "$continue" = 'Y' || "$continue" = 'Yes' || "$continue" = 'YES' || "$continue" = '' ]]
		then :
		else echo exiting.
		exit 0
	fi
	else :
fi
emerge $parameter world > $emergelist
errorcheck
emerge $parameter world
echo -e "\nThese are the packages that will be installed.  If you want to alter this list,\njust edit the file 'emergelist' in $homedir. When finished,\nrun 'update-world --install'"
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' | sort -d > $emergetemp
cp $emergetemp $emergelist
rm -f $failedlist
}
#### end prepare ####



#### installworld ####
# Install the packages listed in emergelist
installworld ()
{
# Make sure there's an emergelist
if [[ -s "$emergelist" ]]
	then :
	else echo "WARNING: no 'emergelist' exists.  Did you run 'update-world --prepare' first?"; exit 1
fi

errorcheck
condition=false
# set begin time
before_time="${before_time:=`date +%s`}"
# Until loop to make portage continue until finished
until [[ $condition = true ]]
do
	cat $emergelist | xargs emerge -1
	# Detect which packages have been successfully emerged and remove from the queue
	installed=`tac /var/log/emerge.log | sed '/Started\ emerge/,$d' | grep "completed emerge" | sed 's/^.*)\ //' | sed 's/\ to.*//'`
	for each in `cat "$emergelist" | sed 's/^=//'`
	do
		if [[ -n `echo "$installed" | grep "$each"` ]]
			then
			# Checking emerge logs for any ebuild notices
			einfo=""
			notice="$noticedir/${each/\//:}-notice.log"
			elog="${PORT_LOGDIR}/elog/`ls -lt $PORT_LOGDIR/elog | grep ${each/\//:} | sed '2,$ d' | sed 's/.*\ //'`"
			if [[ -n "$PORTAGE_ELOG_SYSTEM" ]] && [[ -n "${einfo:=`cat $elog | sed '/^$/d' | sed '/^[ ]*$/d' | sed 's/.*/ * &/'`}" ]]
				then 
				echo ------------------------ > $notice
				echo -e "$each \n" >> $notice
				echo "$einfo" >> $notice
				echo ------------------------ >> $notice
				else :
			fi
			# Removing successful emerges 
			each=`echo ${each//\//\\/} | sed 's/\-[0-9].*//'`
			cat $emergelist | sed "/${each//\//\\/}/d" > $emergetemp
			mv $emergetemp $emergelist
		fi
	done
	# See if all packages have been emerged.  If so, exit function "installworld"
	if [[ -z `cat "$emergelist"` ]]
		then condition=true
		break
		else :
	fi
	## If there are remaining packages, find why the emerge stopped
	errorlog=`ls -t $PORT_LOGDIR | sed '2,$d'`

	# if user-initiated "control-c" just exit.
	if [[ -n `cat $PORT_LOGDIR/$errorlog | grep 'signal 2'` ]]
		then echo -e "\n*** User hit 'Control-C' ... exiting.\n"
		exit 1
		else :
	fi

	# if package failed, give user chance to stop update-world
	sleep 5
	echo
	echo "*** Hit Control-C to exit, or just wait to continue with emerge."
	echo
	sleep 10
	echo "*** Continuing with emerge."

	# Detect failed emerge and add to failedlist, remove failed package from emergelist
	failedpkg=`tac /var/log/emerge.log | sed '1d' | sed '/terminating/,$d' | sed '/completed\ emerge/,$d' | grep ">>>" | sed 's/.*)\ //' | sed 's/\ to.*//'`
	#### uh-oh, stop if something's really wrong to prevent infinite emerge (tried to emerge same package again)
	if [[ -n `tail -n 1 $failedlist | grep $failedpkg` ]]
		then echo "The failed package $failedpkg could not be merged and must be successfully installed before continuing."
		exit 1
		else :
	fi
	
	echo $failedpkg >> $failedlist
	# if package added to emergelist manually doesn't have a version, make sure it's still removed if it fails
	if [[ -z `cat "$emergelist" | grep "$failedpkg"` ]]
		then failedpkg2=`equery which =$failedpkg | sed 's/\/[^/]*$//' | sed -n 's/.*\/\([^/]*\/[^/]*\)/\1/p'`
		cat $emergelist | sed "/^${failedpkg2//\//\\/}/d" > $emergetemp
		else cat $emergelist | sed "/${failedpkg//\//\\/}/d" > $emergetemp
	fi
	mv $emergetemp $emergelist
	# link to the log file of the failed package
	ln -s $PORT_LOGDIR/$errorlog $logdir/${errorlog}_error-log
	echo
	echo
	echo "*** $failedpkg compile failed, skipping."
	# Is there a package in 'emergelist' that REQUIRES the failed package as a dep? If so, remove it too.
	failedpkgdeps=`equery which =$failedpkg | sed 's/\/[^/]*$//' | sed -n 's/.*\/\([^/]*\/[^/]*\)/\1/p'`
	deps=`equery depends $failedpkgdeps | sed '/^\[/d' | sed 's/.*/=&/'`

	for each in `echo $deps`
	do
		if [[ -n `cat $emergelist | grep "$each"` ]]
			then
			if [[ -n `emerge -p $each | grep "$failedpkg"` ]]
				then
				each2=`cat $emergelist | grep $each | sed 's/\=//'`
				echo "$each2 (depends on $failedpkg)" >> $failedlist
				cat $emergelist | sed "/${each//\//\\/}/d" > $emergetemp
				mv $emergetemp $emergelist
				echo "*** $each depends on $failedpkg, skipping."
				else :
			fi
			else :
		fi
	done
	if [[ -z `cat "$emergelist"` ]]
		then condition=true
		break
		else :
	fi
	echo "*** Continuing with emerge world"
done
}
#### end installworld ####

#### revdep-rebuild ####
# this script can call revdep-rebuild and get it's emergelist from there.
# useful to run in conjunction with update-world --install, it will automatically
# run revdep-rebuild after the emerge -uD world is complete
revdep ()
{
rm /root/.revdep-rebuild*.?_*
revdep-rebuild --package-names --pretend | tee $emergelist
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' > $emergetemp
mv $emergetemp $emergelist
}
#### end revdep-rebuild ####

#### piped ####
# emergelist is piped from another emerge (i.e., emerge kde | update-world)
piped ()
{
echo "$piped" > $emergelist
errorcheck
cat $emergelist | sed -n 's/^\[ebuild[[:upper:][:lower:] ]*\]\ \([[:lower:][:upper:]0-9._+\/-]*\).*/\1/p' | sed 's/.*/=&/' > $emergetemp
mv $emergetemp $emergelist
cat $emergelist | xargs emerge -p
echo -e "\nThese are the packages that will be installed.  If you want to alter this list,\njust edit the file 'emergelist' in $homedir. When finished, run\n'update-world --install'"
}
#### end piped ####

#### finished ####
# Display closing messages
finished ()
{
echo -e "\n\nCongratulations!  'update-world' complete."
elapsed_time
echo
# All done! display the failed packages, if any
if [[ -z `cat $failedlist` ]]
	then echo "All packages emerged successfully"
	else echo "These packages couldn't be merged due to compile errors:"
	echo
	cat $failedlist
	echo
	echo -e "Look in $logdir/ for the portage log files of the failed builds.\nCheck bugs.gentoo.org or the Gentoo Forums for help."
fi
if [[ -n "$PORTAGE_ELOG_SYSTEM" ]]
	then echo -e "\n * To review portage elog notices, type \"update-world --log\"\n"
	else echo " * To enable portage notice viewing with elog, check out http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=1#doc_chap4."
fi
}
#### end finished ####

#### elapsed_time ####
# just for fun let's see how long the process takes...
elapsed_time ()
{
after_time=`date +%s`

time=$(( after_time - before_time ))

days=$(( time / 86400 ))
time=$(( time - ( days * 86400 ) ))
hours=$(( time / 3600 ))
time=$(( time - ( hours * 3600 ) ))
minutes=$(( time / 60 ))
time=$(( time - ( minutes * 60 ) ))
seconds=$time

if [[ "$days" -eq 0 ]]
	then days=""
elif [[ "$days" -eq 1 ]]
	then days="$days day, "
elif [[ "$days" -gt 1 ]]
	then days="$days days, "
fi

if [[ "$hours" -eq 0 ]]
	then hours=""
elif [[ "$hours" -eq 1 ]]
	then hours="$hours hour, "
elif [[ "$hours" -gt 1 ]]
	then hours="$hours hours, "
fi

if [[ "$minutes" -eq 0 ]]
	then minutes=""
elif [[ "$minutes" -eq 1 ]]
	then minutes="$minutes minute, and "
elif [[ "$minutes" -gt 1 ]]
	then minutes="$minutes minutes, and "
fi

seconds="$seconds seconds"

echo -e "Total compiling time: $days$hours$minutes$seconds.\n"
}
#### end elapsed_time ####

#### What should the script do when invoked? ####
# I have nine functions: displayhelp, empty, prepare, errorcheck, installworld, revdep, piped, elapsed_time, and finished

arguments="$@"

# update-world --help
if [[ -n `echo "$arguments" | sed -n 's/-h/&/p'` || -n `echo "$arguments" | sed -n 's/--help/&/p'` ]]
	then
	displayhelp
	exit 0
	else :
fi

# update-world --empty
if [[ -n `echo -e \\t "$arguments" | sed -n 's/-e/&/p'` || -n `echo "$arguments" | sed -n 's/--empty/&/p'` ]]
	then echo Preparing list of packages...
	empty
	prepare
	exit 0
	else :
fi

# update-world --prepare
if [[ -n `echo "$arguments" | sed -n 's/-p/&/p'` || -n `echo "$arguments" | sed -n 's/--prepare/&/p'` ]]
	then echo Preparing list of packages...
	prepare
	exit 0
	else :
fi

# update-world --install
if [[ -n `echo "$arguments" | sed -n 's/-[[:lower:]]*i/&/p'` || -n `echo "$arguments" | sed -n 's/--install/&/p'` ]]
	then
	installworld
	check=true
	else :
fi

# update-world --revdep
if [[ -n `echo "$arguments" | sed -n 's/-[[:lower:]]*r/&/p'` || -n `echo "$arguments" | sed -n 's/--revdep/&/p'` ]]
	then
	revdep
	if [[ -s "$emergelist" ]]
		then installworld
		else :
	fi
	check=true
	else :
fi

# emerge -p <package> | update-world
# piped command MUST include --pretend !!!
if [[ "$arguments" = "" ]]
	then echo "No arguments supplied.  If you're piping from the emerge command (emerge -p <package> | update-world), this is fine.  Otherwise, cancel and type 'update-world --help' for usage information."
	until [[ `echo "$piped" | wc -l` -eq 10 ]] && [[ -z `echo "$piped" | grep 'Calculating dependencies'` ]] || [[ -z `echo "${piped:-1}" | tail -n 4` ]]
	do read line
	piped="${piped}
$line"
	done
	else :
fi

if [[ -n `echo "$piped" | sed -n 's/Calculating\ dependencies/&/p'` ]]
	then
	piped
	exit 0
	else :
fi

# update-world --log
if [[ -n `echo "$arguments" | sed -n 's/-l/&/p'` || -n `echo "$arguments" | sed -n 's/--log/&/p'` ]]
then echo "Choose a date: "
select directory in `ls -Al | grep ^d | sed 's/.*\ //'`
do 
	if [[ -n "$directory" ]]
		then date=$directory
		break
		else echo "invalid selection"
	fi
done
rm -f $homedir/$date/notice/log
for each in `ls $homedir/$date/notice/`
do cat $homedir/$date/notice/$each >> $homedir/$date/notice/log
done
less $homedir/$date/notice/log
check=true
exit 0
else :
fi

# Check if arguments were valid
if [[ ${check:-false} == false ]]
	then echo -e "Error: Invalid Command.\nRun 'update-world --help' for usage."
	exit 1
	else :
fi

# done! Display closing text
finished
exit 0

Let me know if you find any bugs or have feedback. Good luck!
[Update] 9/13/06 (version 1.1)
Fixed script to prevent errors when adding a package without specifying a version number to 'emergelist'.
[Update] 10/27/06 (version 1.2)
Script now automatically detects the portage log directory instead of just assuming /var/log/portage.
[Update] 11/22/06 (version 1.4) I guess my edit didn't take yesterday for v1.3...
Added --newuse and --oneshot to emerge variables.
[Update] 11/22/06 (version 1.5)
Now handles 'fetch restriction' properly, prompting user to download sources before continuing.
[Update] 12/06/06 (version 1.6)
Script preserves colored emerge output. Fixed version numbering in help output.
[Update] 12/09/06 (version 1.7)
Cleaned up the code a little, fixed some bugs. Check out the new feature for managing USE flags!
Edit: seems to be broken
[Update] 12/10/06 (version 1.6.1)
bugfixes and utf-8 support.
[Update] 2/16/07 (version 1.
Cleaned and modularized code. Added ability to utilize revdep-rebuild if wished. Can now pipe other
emerges into update-world. Sources /etc/make.conf. Added interfacing with elog. Some bugfixes.

[therealkurisu]

I'm fairly new to Gentoo (and linux in general ^_^) but I find this script to be VERY useful. I've ran it 5 or so times so far and haven't had any problems at all. It saves SO much time from compiling errors and general problems (and allows for skipping updates on known problematic areas!).

Automated skipping, ftw. I reccomend that everyone try it out.

Good job count_zero.


(Also, <3 Crono and therefore, through association, you )

[Doogman]

This looks like a good idea.

While my dual-core system can compile packages at a good clip, rebuilding world always stalls out at some point requiring user intervention. Instead of grinding away at the build list while I'm at work, the computer spends a good deal of time at idle until I get home again. With this script, most of the work should be done while I'm away.

Maybe I'll finally tackle that gcc upgrade I've been putting off.

[devsk]

Removing deps of the failed pkgs is definitely better idea than portage's 'emerge --resume --skipfirst' which can break your system in no time.

[count_zero]

It was discussed in another thread about combining this script with emerging packages in the correct order, a la this thread by Guenther Brunthaler (thanks for a job well done, Guenther ).

If you run Guenther's script up until the point at which you will start rebuilding your system (gcc already emerged), you'll end up with a file '/root/recompile-remaining-packages'. This file contains all of the packages to be emerged, in the order to best rebuild the toolchain and the rest of the system. It's pretty easy to interface this with update-world:

Code: Select all

grep "item " /root/recompile-remaining-packages | sed 's/item\ .*\ //' > ~/.update-world/emergelist

and then

Code: Select all

update-world --install

Have fun with interruption-free emerging of your system in the "correct" order.

[Bynds]

This script has just cured every headache I had with my Gentoo box (I'm not a heavy user )
It took me weeks to update my old compaq server when I dusted it off after about 2 years in the cupboard, this would have saved me about half my time.

Fantastic!

[bender02]

Great script, thanks!

I have one thing though, while running "update-world --install", I get an error like

Code: Select all

cat: /var/log/portage/elog: Is a directory

and the dir inside .update-world which is supposed to contain the logs for failed packages contains only a link to /var/log/portage/elog.
My logs structure for elog is that /var/log/portage/elog is a directory containing logs for various emerges. Is there an easy way to fix this? Thanks.

[count_zero]

The problem was that the script assumed that /var/log/portage was the portage log directory. I've updated the script so that it detects the setting of PORT_LOGDIR in /etc/make.conf. I've updated the code in the first post, but my (free) webhost is giving me some problems so I can't update the download yet

[bender02]

Great! Thanks! (But I'm gonna get to test it only next time I sync...)

[count_zero]

Download link fixed!

[ryker]

This script needs more exposure! This is the most useful emerge wrapper that I have seen yet. They should feature it on GWN sometime.

Thank you for your time and effort on this.

[richfish]

Shouldn't this script use --oneshot for merging packages so it doesn't pollute the world file with all the packages it updates?

[count_zero]

Good idea!

Easy to implement too. I just added '-1' to the end of line 101 in the script.

[RageOfOrder]

Wow, this is just awesome. I've been putting this off for weeks. I think I'll finally update everything on wednesday night.

Thanks for all your hard work!

[dkure]

It would be great if there was two more features.

This script should include --newuse, to make sure use flags are respected.

Second is more of a request, and is a package fetch option. This would fetch the packages selected in the .update_world file. Allowing packages to be downloaded in the background overnight before packages are selected if need be.

I agree that this script should certainly get featured on GNN, makes updating simpler and more specialized to peoples needs.

[count_zero]

This script should include --newuse, to make sure use flags are respected.

I guess the meaning of "--newuse" has changed since I last read the man page...I thought that (when used in conjunction with --update world) it updated ONLY those packages that would be affected by recently changed USE flags.
From the Gentoo Wiki (http://gentoo-wiki.com/MAN_emerge):

--newuse (-N)
Tells emerge to include installed packages where USE flags have changed since compilation. An asterisk marks when a USE flag has changed since the package was compiled.

However, the gentoo handbook suggests that it might be useful in better updating an entire system.
From the installation handbook (http://www.gentoo.org/doc/en/handbook/h ... t=2&chap=1):

Code Listing 14: Updating your entire system

# emerge --update --deep world


Since security updates also happen in packages you have not explicitly installed on your system (but that are pulled in as dependencies of other programs), it is recommended to run this command once in a while.

If you have altered any of your USE flags lately you might want to add --newuse as well. Portage will then verify if the change requires the installation of new packages or recompilation of existing ones:

So i guess I'm a little confused about whether --newuse would be helpful in this situation, where all packages will be updated anyway. Doesn't portage look a new use flags anyway? Maybe someone could clarify this? Implementing this in the script is pretty trivial.

Second is more of a request, and is a package fetch option. This would fetch the packages selected in the .update_world file. Allowing packages to be downloaded in the background overnight before packages are selected if need be.

I'll look into this, parallel fetching would be a useful feature. I believe it is going to be built into portage sometime in the future, but it could probably be implemented here. I'll let you know.

I agree that this script should certainly get featured on GNN, makes updating simpler and more specialized to peoples needs.

Thanks for all the helpful feedback everyone, it makes doing stuff like this very enjoyable.

[ryker]

I believe --newuse also implies --update. Of course, you can always add additional parameters such as -D to catch more packages. I added --newuse to the update-world script on my system and it works fine.

I just changed

Code: Select all

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD'

to

Code: Select all

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD --newuse'

I love this script so much, I was planning on writing and submitting a tip to the GWN this weekend.

[padoor]

this is looking good for me too.
please instruct how do i install and run this .
which script do i download and how i can use it?
i need codes for full proceedure.
i am still newbee for these operations.

[ryker]

It's pretty straight forward and the instructions are clearly given in the first post.

Basically, all you have to do is:

• 1. Download the latest version of the script from here. and save it as /usr/bin/update-world.
  2. chmod +x /usr/bin/update-world

That's it. Now just type update-world and it tells you what to do.

[padoor]

that was pretty easy
and it works noww.
i am due for world update in 2 days.
i will post the result , thanks for direction ryker

[padoor]

i did an emerge --sync
then emerge -Du systaem.
update-world -p made a file list as follows
localhost ramaswamy # update-world -p
The file 'emergelist' already exists and will be overwritten.
Continue anyway? (y/n) y

These are the packages that would be merged, in order:

Calculating world dependencies ..... ..... ..
!!! Ebuilds for the following packages are either all
!!! masked or don't exist:
net-im/ymessenger

... done!
[ebuild U ] dev-java/sun-jdk-1.5.0.09-r1 [1.5.0.08] USE="X alsa -doc -examples -jce -nsplugin" 32,469 kB
[ebuild N ] virtual/jdk-1.5.0 0 kB
[ebuild U ] x11-themes/gnome-backgrounds-2.16.2 [2.16.1] USE="-debug" 1,172 kB
[ebuild U ] dev-java/sun-jre-bin-1.5.0.09-r1 [1.5.0.08-r1] USE="X alsa -nsplugin" 0 kB
[ebuild U ] dev-cpp/gtkmm-2.10.2-r1 [2.8.3] USE="-accessibility% -debug -doc% -examples%" 0 kB
[ebuild FU ] dev-java/sun-jdk-1.4.2.13 [1.4.2.12-r2] USE="X alsa -doc -examples -jce -nsplugin" 35,509 kB
[ebuild U ] www-client/mozilla-firefox-bin-2.0 [1.5.0.7] LINGUAS="-ar -bg% -ca -cs -da -de -el -en_GB -es -es_AR -es_ES -eu% -fi -fr -fy% -fy_NL% -ga -ga_IE -gu% -gu_IN% -hu -it -ja -ko -lt% -mk -mn% -nb -nb_NO -nl -nn% -nn_NO% -pl -pt% -pt_BR -pt_PT% -ru -sk -sl -sv -sv_SE -tr -zh% -zh_CN -zh_TW (-he%) (-ro%)" 0 kB
[ebuild U ] gnome-base/gnome-vfs-2.16.3 [2.16.2] USE="hal ipv6 ssl -avahi -debug -doc -gnutls -samba" 1,855 kB
[ebuild U ] gnome-base/gnome-desktop-2.16.2 [2.16.1] USE="-debug -doc" 1,202 kB
[ebuild U ] gnome-base/eel-2.16.3 [2.16.1] USE="X -debug" 657 kB
[ebuild U ] gnome-base/nautilus-2.16.3 [2.16.2-r1] USE="X gnome -beagle -debug" 4,168 kB
[ebuild U ] gnome-extra/evolution-data-server-1.8.2 [1.8.1-r1] USE="ipv6 ldap ssl -debug -doc -kerberos -keyring -krb4 -nntp" 6,800 kB
[ebuild U ] gnome-extra/nautilus-cd-burner-2.16.2 [2.16.1-r1] USE="cdr dvdr -debug" 698 kB
[ebuild U ] media-libs/xine-lib-1.1.3_pre20061119 [1.1.3_pre20061112] USE="X aac aalib alsa arts dvd esd gnome gtk ipv6 mad nls opengl oss sdl truetype vorbis win32codecs xv -a52 (-altivec) -debug -directfb -dts -dxr3 -fbcon -flac -imagemagick -libcaca -mmap -mng -modplug -pulseaudio -samba -speex -theora -v4l -vcd -vidix -xinerama -xvmc" 6,814 kB
[ebuild U ] gnome-base/control-center-2.16.2 [2.16.1] USE="alsa eds -debug" 1,975 kB
[ebuild U ] gnome-extra/gnome-games-2.16.2 [2.16.1.1] USE="-artworkextra -debug -guile (-avahi%)" 5,134 kB
[ebuild U ] www-client/epiphany-2.16.2 [2.16.1] USE="python -debug -doc" 4,130 kB
[ebuild U ] media-sound/sound-juicer-2.16.2 [2.16.1] USE="ogg -debug -flac" 1,236 kB
[ebuild U ] app-arch/file-roller-2.16.2 [2.16.1] USE="gnome%* -debug" 1,105 kB
[ebuild U ] gnome-extra/gtkhtml-3.12.2 [3.12.1] USE="-debug -static" 1,255 kB
[ebuild U ] mail-client/evolution-2.8.2.1 [2.8.1.1] USE="crypt dbus hal ipv6 ldap spell ssl -bogofilter -debug -doc -kerberos -krb4 -mono -nntp -pda -profile" 12,618 kB
[ebuild U ] net-misc/knemo-0.4.6 [0.4.5] USE="arts -debug -wifi -xinerama" 589 kB
[ebuild U ] kde-base/nsplugins-3.5.5-r1 [3.5.5] USE="arts -debug -kdeenablefinal -kdehiddenvisibility -xinerama" 0 kB

Total: 23 packages (22 upgrades, 1 new), Size of downloads: 119,393 kB
Fetch Restriction: 1 package (1 unsatisfied)

These are the packages that will be installed. If you want to alter this list,
just edit the file 'emergelist' in /root/.update-world. When finished, run
'update-world --install'
localhost ramaswamy #

localhost ramaswamy # update-world --install
Calculating dependencies
emerge: there are no ebuilds to satisfy "=Fetch".

cat: /wget-log.3: No such file or directory

*** Hit Control-C to exit, or just wait to continue with emerge.

*** Continuing with emerge.
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
sed: -e expression #1, char 0: no previous regular expression
ln: creating symbolic link `/root/.update-world/2006-11-23/log/wget-log.3_error-log': File exists


*** compile failed, skipping.
grep: invalid option -- ,
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unrecognized option `--all-packages'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown directories method
grep: option `--direct' requires an argument
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown devices method
grep: unrecognized option `--indirect'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
*** Continuing with emerge world


Usage:
emerge [ options ] [ action ] [ ebuildfile | tbz2file | dependency ] [ ... ]
emerge [ options ] [ action ] < system | world >
emerge < --sync | --metadata | --info >
emerge --resume [ --pretend | --ask | --skipfirst ]
emerge --help [ system | world | config | --sync ]
Options: -[abBcCdDefgGhikKlnNoOpqPsStuvV] [--oneshot] [--newuse] [--noconfmem]
[ --color < y | n > ] [ --columns ]
[--nospinner]
[ --deep ] [--with-bdeps < y | n > ]
Actions: [ --clean | --depclean | --prune | --regen | --search | --unmerge ]


For more help try 'emerge --help' or consult the man page.

Congratulations! 'emerge world' complete.

All packages emerged successfully
localhost ramaswamy #

is this my options error or the fetch has some problem?
regular emerge -Du world does fetch the files and started with jdk

what mistake i am doing??

[count_zero]

I believe --newuse also implies --update. Of course, you can always add additional parameters such as -D to catch more packages. I added --newuse to the update-world script on my system and it works fine.

I just changed

Code: Select all

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD'

to

Code: Select all

# parameter for '--empty' to emerge -e world
if [[ $1 == '--empty' || $1 == '-e' ]]
   then parameter='-pev'
   else parameter='-puvD --newuse'

Done!
I tested it also, and seems to work well.
Version 1.4 now reflects these changes. The script above and the download link are updated.

[count_zero]

@padoor:

you've uncovered a couple of problems, I think they are pretty easily fixed.

Calculating world dependencies ..... ..... ..
!!! Ebuilds for the following packages are either all
!!! masked or don't exist:
net-im/ymessenger

Portage isn't finding this package for your system. It's probably masked on your system, and *could* (though maybe it shouldn't, find out why it's masked) be unmasked by adding 'net-im/ymessenger' to /etc/portage/package.unmask.

[ebuild FU ] dev-java/sun-jdk-1.4.2.13 [1.4.2.12-r2] USE="X alsa -doc -examples -jce -nsplugin" 35,509 kB
. . .
Fetch Restriction: 1 package (1 unsatisfied)

You'll need to download the sources for sun-jdk from the website and copy to /usr/portage/distfiles/ before running the script.
Fix these problems before re-running update-world.

I should be able to make some changes to the script so it handles these errors better.

[padoor]

i will finish emerge of jdk and stop the emerge -Du world and try again.
ymessenger is full masked and i have the latest version installed. so it can skip this file
lets see after i finish this jdk emerge,

[count_zero]

The script version 1.5 (second one tonight ) properly handles "Fetch Restriction" now, prompting you to download the sources before continuing.