[HOWTO] Hassle-free emerge -uD world (update-world v1.8)

[ryker]

Thanks. Did you think any about integrating revdep-rebuild into the script? I haven't really looked into how, it would just be cool to have only one script to run to keep things humming along.

I don't know that integrating revdep-rebuild would be much more than tacking it on the end of this script and adding an option to invoke it. Since that is the case, why not just create your own little script that first calls the update-world script and then revdep-rebuild afterwards?

What I would much rather see, is this update-world script integrated with the wonderful emwrap script. That would be great. So, not only would it be easier to update the world, when it was done, the tool chain would be properly built as well. And, with the least amount of compiling necessary. Obviously, this is a huge request and would take poor count_zero a sizable chunk of his free time.

[lngndvs]

I think I fell into a pit: when a power outage sent my machine South, upon restarting (having already installed some 80% of the packages on the list), update-world --install started from the very beginning of the iist.

I know, I know: I should invest in an UPS, but I've seldom come across any problems (that I knew about) restarting after a power outage. We have alot of them in my location.

Alan

[ryker]

Just a guess, but I would think that another update-world --pretend followed by update-world --install should work fine. It should ask you if you want to overwrite the existing emerge list, which you would say yes to. If not, you can always just delete your .update-world folder and try again.

[drjimmy42]

Thanks. Did you think any about integrating revdep-rebuild into the script? I haven't really looked into how, it would just be cool to have only one script to run to keep things humming along.

I don't know that integrating revdep-rebuild would be much more than tacking it on the end of this script and adding an option to invoke it. Since that is the case, why not just create your own little script that first calls the update-world script and then revdep-rebuild afterwards?

I did. And it works fine. But while we're all here fawning over a cool script that makes our lives easier, why not make everyone's life easier by actually having all your libraries consistent when you think you are done merging. I personally think that revdep should be built into emerge as it seems silly to me to allow emerge to look as if it finished successfully while silently breaking lots of other things. The next best thing might be to have it in a cool script like this that takes care of upgrading for you from end to end.

[count_zero]

Version 1.7 is released which features USE flag management. When the emergelist is created in the first step, a list of USE flags will show up next to the package and can be turned on and off with by adding or removing '-'. It doesn't work for slotted builds yet, but otherwise it works pretty well for me. Test it and let me know any problems.

[BigMichi1]

i recently loaded the 1.7 version of the script but now i got this when running "update-world -p" followed by "update-world -i"

Code: Select all

bigmichi1 tool # ./update-world -p

These are the packages that would be merged, in order:

Calculating world dependencies... done!

[ebuild     U ] sys-apps/portage-2.1.2_rc3 [2.1.2_rc2-r5] USE="doc epydoc -build (-selinux)" LINGUAS="-pl" 328 kB
*** Portage will stop merging at this point and reload itself,
    then resume the merge.
[ebuild     U ] app-arch/gzip-1.3.8 [1.3.7] USE="nls pic -static" 533 kB
[ebuild     U ] media-libs/libsamplerate-0.1.2-r1 [0.1.2] USE="fftw sndfile" 0 kB
[ebuild     U ] net-libs/gnutls-1.6.0 [1.4.4-r1] USE="doc nls zlib" 4,127 kB
[ebuild   R   ] kde-base/kopete-3.5.5-r2  USE="addbookmarks alias arts autoreplace connectionstatus contactnotes crypt gadu groupwise highlight history irc jingle kdeenablefinal* kdehiddenvisibility latex netmeeting nowlistening sametime* slp sms ssl statistics texteffect translator webpresence winpopup xinerama xscreensaver yahoo -debug" 0 kB

Total: 5 packages (4 upgrades, 1 reinstall), Size of downloads: 4,986 kB


These are the packages that will be installed.  If you want to alter this list,
just edit the file 'emergelist' in /root/.update-world. When finished, run
'update-world --install'
bigmichi1 tool # cat /root/.update-world/emergelist
=app-arch/gzip-1.3.8  USE="nls pic -static"
=kde-base/kopete-3.5.5-r2  USE="addbookmarks alias arts autoreplace connectionstatus contactnotes crypt gadu groupwise highlight history irc jingle kdeenablefinal* kdehiddenvisibility latex netmeeting nowlistening sametime* slp sms ssl statistics texteffect translator webpresence winpopup xinerama xscreensaver yahoo -debug"
=media-libs/libsamplerate-0.1.2-r1  USE="fftw sndfile"
=net-libs/gnutls-1.6.0  USE="doc nls zlib"
=sys-apps/portage-2.1.2_rc3  USE="doc epydoc -build"
bigmichi1 tool # ./update-world -i
Checking use flags.....done!
WARNING: no 'emergelist' exists.  Did you run 'update-world --prepare' first?
bigmichi1 tool #

some help would be nice, i like this sript

[thomasvk]

Hey I haven't tried the script yet (I'm not that kind of user that starts emerge and goes away, I just leave an ssh session open to watch. ) but I noticed something and I just wanted to give a suggestion. It's about:

The script version 1.5 (second one tonight ) properly handles "Fetch Restriction" now, prompting you to download the sources before continuing.

If the use of this script is to just run along at one go, wouldn't it be better to skip packages with fetch restriction if the files are absent (if they are present they can just be installed) and then install those at the end or just don't install them and in the logs make note that you need to download them first, or something so it won't hold up the script. Or is this already what's happening?

Out of interest I will surely try this script next time I'm updating. Good work.

[count_zero]

i recently loaded the 1.7 version of the script but now i got this when running "update-world -p" followed by "update-world -i"

Code: Select all

bigmichi1 tool # ./update-world -p

These are the packages that would be merged, in order:

Calculating world dependencies... done!

[ebuild     U ] sys-apps/portage-2.1.2_rc3 [2.1.2_rc2-r5] USE="doc epydoc -build (-selinux)" LINGUAS="-pl" 328 kB
*** Portage will stop merging at this point and reload itself,
    then resume the merge.
[ebuild     U ] app-arch/gzip-1.3.8 [1.3.7] USE="nls pic -static" 533 kB
[ebuild     U ] media-libs/libsamplerate-0.1.2-r1 [0.1.2] USE="fftw sndfile" 0 kB
[ebuild     U ] net-libs/gnutls-1.6.0 [1.4.4-r1] USE="doc nls zlib" 4,127 kB
[ebuild   R   ] kde-base/kopete-3.5.5-r2  USE="addbookmarks alias arts autoreplace connectionstatus contactnotes crypt gadu groupwise highlight history irc jingle kdeenablefinal* kdehiddenvisibility latex netmeeting nowlistening sametime* slp sms ssl statistics texteffect translator webpresence winpopup xinerama xscreensaver yahoo -debug" 0 kB

Total: 5 packages (4 upgrades, 1 reinstall), Size of downloads: 4,986 kB


These are the packages that will be installed.  If you want to alter this list,
just edit the file 'emergelist' in /root/.update-world. When finished, run
'update-world --install'
bigmichi1 tool # cat /root/.update-world/emergelist
=app-arch/gzip-1.3.8  USE="nls pic -static"
=kde-base/kopete-3.5.5-r2  USE="addbookmarks alias arts autoreplace connectionstatus contactnotes crypt gadu groupwise highlight history irc jingle kdeenablefinal* kdehiddenvisibility latex netmeeting nowlistening sametime* slp sms ssl statistics texteffect translator webpresence winpopup xinerama xscreensaver yahoo -debug"
=media-libs/libsamplerate-0.1.2-r1  USE="fftw sndfile"
=net-libs/gnutls-1.6.0  USE="doc nls zlib"
=sys-apps/portage-2.1.2_rc3  USE="doc epydoc -build"
bigmichi1 tool # ./update-world -i
Checking use flags.....done!
WARNING: no 'emergelist' exists.  Did you run 'update-world --prepare' first?
bigmichi1 tool #

some help would be nice, i like this sript

That's strange. I just tried it with your packages on my system and it works without problem. Try deleting the emergelist and workdirectory and trying again.

[BigMichi1]

ok there is something i found
the line in the install section

Code: Select all

cat $emergelist | sed 's/\ .*//' > $emergelist

produces an empty list for me

[count_zero]

ok there is something i found
the line in the install section

Code: Select all

cat $emergelist | sed 's/\ .*//' > $emergelist

produces an empty list for me

You're right! I'm not sure why that's even there...
Fixed the code and the download.

@t0maz
You might try version 1.7b to see if it fixes your issue.

[slackline]

Hi,

Thanks for the script, looks like a great idea, so frustrating setting things running before going to bed only to wake up and find that things fell over on package 2 out of 68

Tried it out (v1.7b) and found a conflict with my default emerge options. I have --ask and --verbose set in /etc/make.conf...

Code: Select all

## Default options for emerge
EMERGE_DEFAULT_OPTS="--ask --verbose"

This appears to stop things running as after preparing the install went...

Code: Select all

# update-world -i
Checking use flags...................done!
!!! "--ask" should only be used in a terminal. Exiting.
cat: /dead.letter: No such file or directory

*** Hit Control-C to exit, or just wait to continue with emerge.

*** Continuing with emerge.
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
sed: -e expression #1, char 0: no previous regular expression


***  compile failed, skipping.
grep: invalid option -- ,
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unrecognized option `--all-packages'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown directories method
grep: option `--direct' requires an argument
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: unknown devices method
grep: unrecognized option `--indirect'
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
*** Continuing with emerge world
!!! "--ask" should only be used in a terminal. Exiting.
Congratulations!  'emerge world' complete.

All packages emerged successfully

in about 30 seconds for 27 packages (I wish I had a comp that was that quick!!!).

Is there anyway of checking the global options and removing this flag if its set?

The other options is simply to remove the --ask from my global options, but I'm lazy and can't be bothered using it (or -p) when I want to look at what emerging a package will pull in.

Thanks again for a useful tool,

slack

[count_zero]

Hi,

Thanks for the script, looks like a great idea, so frustrating setting things running before going to bed only to wake up and find that things fell over on package 2 out of 68

Tried it out (v1.7b) and found a conflict with my default emerge options. I have --ask and --verbose set in /etc/make.conf...

Code: Select all

## Default options for emerge
EMERGE_DEFAULT_OPTS="--ask --verbose"

Hmm, this is a portage feature that I didn't even know existed...I'll have to look into it, but I'm sure that if it is possible to disable the default flags from the command line, it's a pretty trivial addition. Maybe something like

Code: Select all

EMERGE_DEFAULT_OPTS="" emerge -uD world

or something similar.

[bunkacid]

@count_zero
Hey, I dont think you should be playing with the USE flags. I felt that the previous version I was using was pretty damn stable.
The latest, 1.7b just hosed the very complex USE flags from my /etc/portage/package.use.
(It would have been nice of you to make a backup file first.)

@users, Lesson for the day, ALWAYS remember to keep a nice pretty backup of your files and configuration settings before making changes and or playing with developing scripts.

Now it's time to restore from a rather old but more complete version of my package.use. Hahaha, I said rather old, but looking at what I have it's only 44 days, could have been much worse.

OI

[count_zero]

Ouch! Sorry for that...I'll take 1.7b down until I can figure what's wrong...
(you can still download it on http://countzero.amidal.com/files/update-world-1.7b)

[mci_nano]

Hi...

It seems not working for packages without USE Flags... they are not in emergelist because of

Code: Select all

sed -n 's/.*\(^.*USE=\"[a-zA-Z0-9_\ *%()-]*\"\).*/\1/p'

I actualy dont have time to fix that but wanted to point you to it... my examples are

Code: Select all

[ebuild     U ] mail-client/mozilla-thunderbird-bin-1.5.0.8 [1.0.7] 10,368 kB 
[ebuild     U ] www-client/mozilla-firefox-bin-1.5.0.8 [1.5.0.3] LINGUAS="de%* -ar% -ca% -cs% -da% -el% -en_GB% -es% -es_AR% -es_ES% -fi% -fr% -ga% -ga_IE% -he% -hu% -it% -ja% -ko% -mk% -nb% -nb_NO% -nl% -pl% -pt_BR% -ro% -ru% -sk% -sl% -sv% -sv_SE% -tr% -zh_CN% -zh_TW%" 8,419 kB 

And also... maybe you will add utf8 support? as i mentioned in my last post otherwise it wount work for some packages ending with a letter in the version brackets... example [2006o]

ahh, now i see v1.7 is reseted... ok then

[baldo]

perhaps it would be nice to have an comandline-argument to enable/disable use-flag support...

[count_zero]

I fixed a few bugs for a minor 1.6.1 release.
EMERGE_DEFAULT_OPTS set in make.conf shouldn't cause any more options.
I wrote a new way for getting packages out of the raw emerge output, so there shouldn't be any more gibberish like =*** ending up in the emergelist.
There's no USE flag support in this version, hopefully I'll fix 1.7 in the next few days.
Happy testing! (and remember to make backups! works for me, but that doesn't mean it works for you!)

And also... maybe you will add utf8 support? as i mentioned in my last post otherwise it wount work for some packages ending with a letter in the version brackets... example [2006o]

Try this version out, I made some changes which should help utf-8 problems. I can't test utf-8 problems myself, so you'll have to help.

[roseZ]

If you want to do things cleanly, don't try to parse make.conf variables with sed like this:

Code: Select all

portlogdir=`cat /etc/make.conf | grep ^PORT_LOGDIR | sed 's/PORT_LOGDIR=//' | sed 's/\"//g' | sed 's/\/$//'`

Just let bash parse it for you by adding as first line

Code: Select all

source /etc/make.conf

and use PORT_LOGDIR casually after that. Since make.conf should only be used to declare variables, sourcing it ought to be safe.

Compared to using sed, this could save you a lot of hassle, e.g. make.conf might contain

Code: Select all

PORT_LOGDIR="${SOMEDIR}/SUBDIR"
PORT_LOGDIR="/hmm/i/changed/my/mind/and/will/instead/use/this/\
very/long/path/with/line/breaks/in/it"

As to whether the variable should be checked like [post=3767173]capira suggests[/post], let me just refer to make.conf's man page for portage behavior

Cheers

[count_zero]

If you want to do things cleanly, don't try to parse make.conf variables with sed like this:

Code: Select all

portlogdir=`cat /etc/make.conf | grep ^PORT_LOGDIR | sed 's/PORT_LOGDIR=//' | sed 's/"//g' | sed 's/\/$//'`

Just let bash parse it for you by adding as first line

Code: Select all

source /etc/make.conf

and use PORT_LOGDIR casually after that. Since make.conf should only be used to declare variables, sourcing it ought to be safe.

True, this is a much easier way to do it. I'll include it in the next release.

[bunkacid]

If you want to do things cleanly, don't try to parse make.conf variables with sed like this:

Code: Select all

portlogdir=`cat /etc/make.conf | grep ^PORT_LOGDIR | sed 's/PORT_LOGDIR=//' | sed 's/"//g' | sed 's/\/$//'`

Just let bash parse it for you by adding as first line

Code: Select all

source /etc/make.conf

Also by sourcing the make.conf file, users who have their make.conf source other files they will be sourced accordingly.

I myself broke up the portions of my make.conf into separate easier to manage files, which live in /etc/portage/make.conf.d/

[bunkacid]

just had another thought.

Code: Select all

update-world -p
The file 'emergelist' already exists and will be overwritten.
Continue anyway? (y/n) 

since by default just hitting enter here, perhaps, capitalization of Y to signify the default option.

[float-]

a nice addon would be to check if genlop is installed, and if it is, print the estimated compile time.
maybe this fits best when you issue --prepare

btw, tried to download your script from the instructions in the GWN, and got a 404 error, my guess is that you renamed your script,
perhaps you should make a symlink from the last stable version to ./update-world

[count_zero]

btw, tried to download your script from the instructions in the GWN, and got a 404 error, my guess is that you renamed your script,
perhaps you should make a symlink from the last stable version to ./update-world

Link fixed, thanks!

By the way, genlop has never been very accurate at predicting compile times for me, but I'll see how it works...

[roseZ]

Once more about reading portage variables: I just noticed there's a tool that can do it (and more). So instead of sourcing make.conf to get PORT_LOGDIR (along with everything else), you could do

Code: Select all

portlogdir=`portageq envvar PORT_LOGDIR`

which feels a tad better to me. Append 2>/dev/null within backquotes if you don't like the warning message for users not in portage group.

[yamamushi]

Thank you Thank you THANK YOU!!!

This script has made my life 1000x easier, I was just thinking a few weeks back "man, portage would be cooler if it set aside packages that wont compile and moved onto the next ones".