Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS

[Drysh]

I don't think it's this script fault, but I had some problems while using it. Please, take a look: http://forums.gentoo.org/viewtopic-t-496817.html

BTW: I nice side effect of your script: It gave me a list of packages that I emerged to look for the evil one.

[COMKEEN]

I have got an error

Code: Select all

# sh genscript.sh
Decoding attachment - please standby.
genscript.sh: line 7: syntax error near unexpected token `||'
genscript.sh: line 7: `|| die "I need openssl for base64 decoding!" '

I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called). Remove the line feed and put all the commands in one line.

Example:

Change

Code: Select all

emerge --oneshot --update openssl \ 
 || die "I need openssl for base64 decoding!" 

to

Code: Select all

emerge --oneshot --update openssl || die "I need openssl for base64 decoding!" 

HTH,
Christian

[Guenther Brunthaler]

I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called).

Thanks for that more accurate problem description!

I therefore conclude the origin of the problem must be the following: When copying/pasting the script, whitespace has somehow been added after the backslash which was not there in the original script.

In order to fix things, that white space must be manually removed: The backslash must be the very last character in the line! No spaces/tabs are allowed to follow.

Seems your editor somehow screwed up the pasted text by inserting some whitespace after the backslashes.

Remove the line feed and put all the commands in one line.

Originally, this was exactly as the line looked! But as I never allow any line in one of my scripts to get longer than 79 characters, I inserted escaped line feeds in order to break long lines.

For the future, I will try to find a way to avoid the escaped backslashes, in order to make the script even work if an editor unintentionally adds whitespace at the line end.

UPDATE: I have updated the script in the article by an updated version (1.12) of the script which avoids all escaped backslashes. (As an unintended side effect, it looks even more cryptic now. But who cares.)

Please try again with the updated script!

[COMKEEN]

I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called).

Thanks for that more accurate problem description!

I therefore conclude the origin of the problem must be the following: When copying/pasting the script, whitespace has somehow been added after the backslash which was not there in the original script.

In order to fix things, that white space must be manually removed: The backslash must be the very last character in the line! No spaces/tabs are allowed to follow.

Seems your editor somehow screwed up the pasted text by inserting some whitespace after the backslashes.

Actually it is not the text editor's fault, but the browser's fault. I am using Konqueror and when I am copying/pasting your code snippet into a text editor (no matter whether I use a GUI editor like KWrite or a command line editor running in Konsole), there are additional whitespaces inserted at the beginning as well as at the end of each line. When I am copying the script from Firefox, everything seems alright.

To remove the leading and trailing whitespaces, one could use this command:

Code: Select all

$ cat genscript_with_spaces.sh | sed -e "s/^\ //g" | sed -e "s/\ $//g" > genscript.sh

(I am sure that the multiple use of sed could be avoided - but since I don't use it very often, I don't know how.)

[AssociateX]

Nice!

Hey, what about when two versions of the same package are installed? I have dev-libs/lzo-1.08-r1 and dev-libs/lzo-2.02-r1 installed. I did a "emerge --sync && emerge --update --deep --newuse world && revdep-rebuild" and everything is good there. lzo-2.02-r1 is what is installed with an "emerge -u lzo" and kino needs liblzo.so.1 from lzo-1.08-r1. You can see that they are both installed below.

Code: Select all

athlon ~ # emerge -pv lzo

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] dev-libs/lzo-2.02-r1  USE="examples" 0 kB

Total size of downloads: 0 kB
athlon ~ # emerge -pv =lzo-1.08-r1

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] dev-libs/lzo-1.08-r1  0 kB

Total size of downloads: 0 kB

But!

Code: Select all

athlon ~ # grep lzo recompile-remaining-packages
item 537 =dev-libs/lzo-2.02-r1
athlon ~ # 

It looks like lzo-1.08-r1 is installed but will not get recompiled with the script. Am I right? If so, can this be fixed?

Guenther Brunthaler, I just wanted to say that I always enjoy reading your threads and other posts. Thank you for the good reading.

[AssociateX]

Strange, I was just hunting for other cases like that last one that I posted and found another case.

Code: Select all

athlon ~ # ls /var/db/pkg/app-crypt/ | grep gnupg
gnupg-1.4.5/
gnupg-1.9.20-r3/
athlon ~ # cat recompile-remaining-packages | grep gnupg
item 140 =app-crypt/gnupg-1.4.5
item 141 =app-crypt/gnupg-1.9.20-r3

But in this case both installed versions were found by your script. I wonder what I might have done wrong to mess up the other case.

#edit starts here.

I found this:

Code: Select all

athlon ~ # emerge -ep world | grep -e lzo -e gnupg
[ebuild  N    ] app-crypt/gnupg-1.4.5
[ebuild  N    ] app-crypt/gnupg-1.9.20-r3
[ebuild  N    ] dev-libs/lzo-2.02-r1

This must be the method that the script uses to find installed packages although it doesn't really list them all other wise it would have found lzo-1.08-r1 also. Maybe something like the following could be used to generate your list of installed apps, minus the sys-devel/gcc and old sys-kernel/* or what ever other stuff you know doesn't need to be installed again :

Code: Select all

find /var/db/pkg -type d | sed "s/\/var\/db\/pkg\//=/" | grep / | sed "s/\/var\/db\/pkg//"|grep =

By the way I have not read the whole thread if this has already been brought up.

[count_zero]

Guenther, great job!
Check out my script and how I've made it interface with yours for interruption-free compiling of the entire system (in the correct order).

[AssociateX]

...

[Guenther Brunthaler]

But in this case both installed versions were found by your script. I wonder what I might have done wrong to mess up the other case.

My script is based on the output of "emerge --emptytree". It re-orders that output and removes duplicate and unnecessary emerges, but will not add anything to that list.

In your case it seems that Portage thinks both GPG-versions are necessary, but not both of the LZO versions.

This might be possible if you left out the optional "emerge -a --depclean"-step of my guide, which will remove unnecessary packages from your system.

But as emerge --depclean is "known to be broken" (as it's own output says), I did not want to make it a non-optional step of my guide.

So, what can you do?

I suggest the following approach:

• Run

  Code: Select all

  emerge -avuDN world

  to be sure everything is in a consistent state from emerge's point of view.
• Run

  Code: Select all

  emerge --ask --depclean

  . Make sure no packages are suggested for removal which you obviously don't want to be removed. Accept the removal of suggested packages only if you agree. This should remove any unnecessary packages.
• Run

  Code: Select all

  revdep-rebuild

  repeatedly until no more packages can been found containing broken library dependencies

. (revdep-rebuild is part of gentoolkit.)

If you are lucky, your old LZO package should have gone by then.

If it's still there, you can use equery depends to check whether any other package actually depends on the old LZO version.

If you don't find any dependent packages, just remove the old version of the LZO package manually using emerge --unmerge.

In order to be sure, use qpkg before the removal and create a binary archive from the LZO version to be removed. If it later turns out the old version is actually required by anyone, you can easily re-install it.

As a general precaution when removing shared libraries, I would also suggest emerging and running sash, the "Stand-Alone Shell", in a different window before removing the library. (A statically-linked busybox can alternatively be used for the same purpose.) sash is a statically-linked shell which will even continue to run if most other tools and shells won't because of shared library problems. In such cases you can use the already-running sash to fix things even if bash won't work any longer.

[Guenther Brunthaler]

Guenther, great job!
Check out my script and how I've made it interface with yours for interruption-free compiling of the entire system (in the correct order).

I like it!

And I totally agree with your idea of an interruption-free rebuild.

I even think it would be the best to incorporate your great idea on next occasion directly into my script, so people won't have to mess around with different scripts for basically the same purpose.

Of course, your contribution will be honored accordingly in the updated version of the guide!

Until then, I recommend using your script for the purpose of interruption-free rebuilding the entire system in combination with my script.

I am also happy to see that my original guide has undergone several revision since then, adding significant improvements suggested by various people, making it even better.

It should be also clear from this, that it's no longer my guide - it's our guide! I'm just the original author and primary maintainer.

[count_zero]

Guenther, great job!
Check out my script and how I've made it interface with yours for interruption-free compiling of the entire system (in the correct order).

I like it!

And I totally agree with your idea of an interruption-free rebuild.

I even think it would be the best to incorporate your great idea on next occasion directly into my script, so people won't have to mess around with different scripts for basically the same purpose.

Of course, your contribution will be honored accordingly in the updated version of the guide!

Until then, I recommend using your script for the purpose of interruption-free rebuilding the entire system in combination with my script.

I am also happy to see that my original guide has undergone several revision since then, adding significant improvements suggested by various people, making it even better.

It should be also clear from this, that it's no longer my guide - it's our guide! I'm just the original author and primary maintainer.

Sounds good.
If I can help, let me know.

[AssociateX]

lzo is a depend but for some reason it was in the world file. That stinks, my fault. Sorry about that.

[Guenther Brunthaler]

Sounds good.

I have just updated the script and guide as of version 1.15. The script now features interruption-free operation, just as you suggested. Your contribution has been mentioned in the guide.

After the package list has been processed, the script automatically tries to recompile the packages again which failed earlier, hoping it will work then. This process will continue recursively, but care has been taken to avoid an infinite loop.

I hope this new approach will also defeat any issues that might arise from circular dependencies in ebuilds.

Furthermore, I have added a log file which captures screen output.

[AssociateX]

Ok, my problem with lzo is a problem with the ebuild. I filed a bug report https://bugs.gentoo.org/show_bug.cgi?id=148266 but it was closed as a duplicate of https://bugs.gentoo.org/show_bug.cgi?id=143939. That bug too was closed but I reopened it, read to see my reason why.

This all brings me back here with some things that I have learned that someone may want to take into consideration with this howto. cd to the dir with the recompile-remaining-packages script and do the following:

To make a sorted list of all packages that are installed:

Code: Select all

find /var/db/pkg -type d|sed "s/\/var\/db\/pkg\//=/"|grep /|sed "s/\/var\/db\/pkg//"|grep =|sort>db.tmp

To make a sorted list of all packages that are recompiled by "emerge -e world":

Code: Select all

emerge -ep world|perl -ne 'if (/]/) { s{^\[[^\]]+\]\s+}{=}; print $_ }'|sort>emerge.tmp

To make a sorted list of all packages that are recompiled by the recompile-remaining-packages script:

Code: Select all

perl -ne 'if (/item [1-9]/) { s{item \d{1,4} }{}; print $_ }' recompile-remaining-packages|sort>recompile.tmp

Now to see the differences.

To list the packages that the recompile-remaining-packages script will not recompile but emerge -e world would:

Code: Select all

diff -bn emerge.tmp recompile.tmp|grep =

For me this gave back:
=sys-devel/gcc-4.1.1
=sys-devel/gcc-config-1.3.13-r3
Pretty much novelty information.

What packages are installed but will NOT be recompiled by "emerge -e world":

Code: Select all

diff -bn emerge.tmp db.tmp|grep =

I got 4 packages listed here that would NOT be recompiled:
=dev-libs/lzo-1.08-r1
=sys-devel/automake-1.4_p6
=sys-devel/automake-1.8.5-r3
=sys-kernel/gentoo-sources-2.6.17-r4
This information is important because these packages are also not recompiled by the recompile-remaining-packages script. Btw, lzo is in that list because of the lack of it's DEPEND in the kino ebuild so I --oneshot it in to keep it out of the world file(as stated above, I already filed a bug about this). I don't know why the other three are not recompiled by emerge -e world. I have 6 versions of automake but only 2 are not recompiled by emerge -e world, and those 2 are not removed by --depclean.

Now I'm off to figure out a one-liner to tell which packages in the world file are dependencies and so may be removed from the world file.

Also, my world file is located at /var/lib/portage/world, not /var/portage/world as this howto has it. Is there a default or even an option to change where the world file is located?

Thank you... and I hope that I'm not being too annoying here.

[paulj]

Hi all,

(You are reading version 1.15 of this guide.)

ABSTRACT: How to recompile your entire system / toolchain with MINIMUM PROCESSING TIME EFFORT.

If you plan to do a major compiler update for your Gentoo box, such as from GCC 3.3 to GCC 3.4, or from GCC 3.3/3.4 to GCC 4, then read on. (This will typically be the case when upgrading your Gentoo installation to the new 2006.1 profile.)

Guenther, you are a star . I have had several trying periods over the last few years with GCC upgrades, and system recompilation, and your script has just made this process much smoother. I am now in the final stages of a full recompilation using your script, and I am sure it has saved me much heartache. Due to my misunderstanding of the changes to the profiles (specifically the need to move from 2006.0 to 2006.1/desktop, not 2006.1) I have had to carry out the compilation operation twice. First time I used the time honoured method, and this time with your script.

THANKYOU!

[Guenther Brunthaler]

Also, my world file is located at /var/lib/portage/world, not /var/portage/world as this howto has it.

You are absolutely right. It's /var/lib/portage/world, not /var/portage/world.

What you have found was clearly a typo in my guide!

Thank you for reporting it!

I have just fixed it as of version 1.16.

[Guenther Brunthaler]

Fixed a rather embarassing bug in version 1.16: The log file was created with the 'minute' number at the position of the 'month' number.

[steveL]

Hi
Firstly, thanks for the script!
I'm using the latest version that you uploaded today. However I get an error (unexpected token >) at line 62 of the recompile-remaining-packages script, which is `exec > >('

Any ideas?

Edit:
Also, is it really necessary to emerge world newuse etc before running the script? Surely these are all going to be compiled anyway?

[Guenther Brunthaler]

I get an error (unexpected token >) at line 62 of the recompile-remaining-packages script, which is `exec > >('

Perhaps you might have been using a very old version of bash? My script assumes a moderately modern version of bash.

The >(...) expression is a process substitition, which will be replaced by the filename of an ephemeral named pipe, which will be connected to the statements in the parentheses. Those statements will be run as a background process, and will read from the pipe; implementing the log file feature.

That is, the

Code: Select all

exec > >(...)

is actually an

Code: Select all

exec > anonymous_pipe

where anonymous_pipe has been constructed on the fly by bash.

However, process substitution is not a feature that all shells provide. For instance, ash does not.

That's why my script starts with

Code: Select all

#! /bin/bash

and does not contain the usual

Code: Select all

#! /bin/sh

Also, is it really necessary to emerge world newuse etc before running the script? Surely these are all going to be compiled anyway?

Yes and no. The problem is, my script assumes a working compiler and tools to be available when it starts its job.

That is, tools like gcc, ld, glibc, make, sed, bison, flex, cp, ar, as, libtool and any other tools involved in the process of an "emerge" must be in a consistent state.

But certainly it will be unnecessary to bring packages like openoffice up to date just for rebuilding the whole system.

However, most people run

Code: Select all

emerge -avuDN world

on a regular basis, which means their systems are close to being up-to-date anyway, and the additional emerge won't hurt.

If you know what you are doing, it is certainly possible to just do a

Code: Select all

emerge --sync

and omit the the emerge -avuDN world step.

But I will not take any responsibility for problems arising from an improperly functioning toolchain, in case the tool chain was in an inconsistent state.

[steveL]

Thank you for the lucid explanation, on both points.

It really is cool to learn something new! I didn't know about anonymous pipes.

I am running bash-3.1_p17; while I don't know if this is old, I doubt it since I only installed this system a month ago. Is there anything else that could cause this?

On the second point, how does the emerge world guarantee a consistent toolchain; surely that's the point of the whole recompile everything metaphor?

Also (minor point) the version given in the generated script is "1.13".

[Guenther Brunthaler]

I didn't know about anonymous pipes.

Well, actually they are not really anonymous from the viewpoint of the kernel, but rather temporary: Bash uses mkfifo or something similar internally to create a pipe with a temporary name in /tmp (I guess), then runs the process substitution, and eventually removes the pipe automatically.

Which means the pipe will actually have some name, but from the viewpoint of the shell script it works like an anonymous pipe. It's also fun and easy to use!

...as long as it works. What seems to be a problem with your system configuration.

I am running bash-3.1_p17; while I don't know if this is old

My bash --version gives "3.1.16(1)-release (i686-pc-linux-gnu)", from which I conclude your bash should be good enough.

So the problem you encountered must be triggered by something else.

Is there anything else that could cause this?

Perhaps it might be easiest to run a few tests manually before going into details how my script works.

First, let's check whether process substitution runs at all in your environment. Try this:

Code: Select all

cat <( echo hello )

This should just print "hello". Note that there must not be a space between the "<" and the "(".

On the second point, how does the emerge world guarantee a consistent toolchain; surely that's the point of the whole recompile everything metaphor?

The idea is very simple: emerge -avuDN world is intended to bring the system up-to-date, including any build tools.

It it doesn't, then Portage has a problem ... and my script will have one also.

Also (minor point) the version given in the generated script is "1.13".

That's OK, the script and the guide have different version numbers. I am referring always to the guide version.

So far, the guide has been much more often updated than the script, and I'm incrementing the version number on each change for both of them independently.

But I realize this might be a cause for confusion; perhaps I'll change this in the future.

BTW: I'm going offline now (it's midnight where I live), so I'm not going to answer any more postings before tomorrow.

[steveL]

My bash --version gives "3.1.16(1)-release (i686-pc-linux-gnu)", from which I conclude your bash should be good enough.
..
Perhaps it might be easiest to run a few tests manually before going into details how my script works.

First, let's check whether process substitution runs at all in your environment. Try this:

Code: Select all

cat <( echo hello )

This should just print "hello". Note that there must not be a space between the "<" and the "(".

Yes it does.

.. the script and the guide have different version numbers. I am referring always to the guide version.

So far, the guide has been much more often updated than the script, and I'm incrementing the version number on each change for both of them independently.

But I realize this might be a cause for confusion; perhaps I'll change this in the future.

Ok, no biggy.

BTW: I'm going offline now (it's midnight where I live), so I'm not going to answer any more postings before tomorrow.

Understood. Thanks for responding!

[Guenther Brunthaler]

Yes it does.

OK, so input redirection works. Now let's check output redirection.

The following script

Code: Select all

#! /bin/bash
exec > >(
	echo "starting output"
	while read LINE; do
		echo "OUTPUT: $LINE"
	done
	echo "stopping output"
)
echo "Hello, world!"
echo "Hello, universe!"

should generate this output:

Code: Select all

starting output
OUTPUT: Hello, world!
OUTPUT: Hello, universe!
stopping output

However, as the output is generated by a background process, it is possible that the shell prompt for the next command might be messed up by the output. This is normal under such conditions.

[steveL]

No, this gives me the same error:
test.sh: line 2: syntax error near unexpected token `>'
test.sh: line 2: ` exec > >( '
Guess we've found the problem (and it's not your script.) Just wish I knew what the sol'n was!

[Guenther Brunthaler]

Guess we've found the problem (and it's not your script.) Just wish I knew what the sol'n was!

I think now it must have something to do with you bash's settings.

You know, the operation of bash is influenced by several environment variables like SHELLOPTS and POSIXLY_CORRECT as well as by builtins such as shopt and set.

I therefore suggest that you check those settings.

For instance, if your bash runs in POSIX mode, it will disable most extensions, including process substitution. Read section 6.11 of the bash info manual for more details on POSIX mode.