Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS

[Guenther Brunthaler]

This script seems very interresting to me, but how much does this approximately take?

It takes bloody ages.

However, it is still the fastest way to recompile your entire system: The other guides (I am aware of) take even bloodier ages...

gcc probably recompiles everything so this is basically a stage1 install?

Yes.

Could I grab a newer gentoo cd and get binary packages for some of the files from that cd and recompile the other files to make it faster?

It depends on whether or not you

1.) Want to use USE flags which are different from the ones used by compiling the Gentoo CD.

2.) Want to use different compiler flags than were used for compiling the Gentoo CD.

For instance, I have an Athlon-1333, and wanted to use the following compiler flags in my make.conf

Code: Select all

CFLAGS="-march=athlon-tbird -O3 -DNDEBUG -pipe -fomit-frame-pointer -fno-stack-check"

in order to fully optimize Gentoo for my machine.

If want you use such settings, which are quite different from the ones used by compiling the Gentoo CD, you have to recompile everything anyway, so using a stage 3 CD will not help.

If you are happy with the defaults used on the CD, however, I certainly would be much faster to make a stage 3 reinstall. (But I have tweaked and twisted my installation so much, that I won't be happy with what has been compiled on the stage3 CD.)

[CPUguy387]

Thanks for the reply.

My system.list generated fine with no apprent errors.

It turns out that in the world.list file, it was showing that ati-drivers and ati-drivers-extra were masked for some odd reason (probably used ACCEPT_KEYWORDS instead of adding it to my package.keywords file!). I added them to my /etc/portage/package.keywords file and then it generated world.list.

I had a block at the top of my list. Pam login was blocking shadow. To fix that, see this thread: http://forums.gentoo.org/viewtopic-t-443022.html, or the short answer, run this command:

To solve "emerge -C pam-login && emerge -1 shadow", this is safe to do

Now my system.list and world.list files are correct, and when I ran your script, it generated the script to recompile everything, so I'm back on track.

Thanks Guenther!
Matt

[jure1873]

It takes bloody ages.

that's what I thought... I'll think a little more before doing this. The problem is that while this is being upgraded it's wise if X is not running, but I need my computer almost every day so I think I'll wait some more time for this. I'll check how much free space do I have for another chroot and how could I grab some stuff from the packages cd.

[Erlend]

I strongly recommend running this script from the text mode interface, without an X-server running. There are too many things that can go wrong if trying to replace an X server by a recompiled version while it is running. (But on the other hand, I have never tried it. Perhaps it may actually work. But I would not count on it.)

I've always recompiled xorg from within xorg and I don't remember having any problems - I'm not saying you won't have problems just that any problems I had were so minor I never noticed.

Why is the script more efficient than just doing "emerge -e world" once after upgrading to gcc 4.1.1?

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

[Guenther Brunthaler]

I've always recompiled xorg from within xorg and I don't remember having any problems - I'm not saying you won't have problems just that any problems I had were so minor I never noticed.

That's good news!

But - what Desktop Environment are you using?

I suspect some environments to be more fragile than others when recompiling them while they are running.

Why is the script more efficient than just doing "emerge -e world" once after upgrading to gcc 4.1.1?

Well - at least it will not recompile GCC a second time

But to be a bit more serious: Just go and consider the output of

Code: Select all

# emerge -ep world | less

Now notice the top entries of the generated list. On my system this are:

Code: Select all

[ebuild  N    ] sys-libs/zlib-1.2.3  USE="-build" 415 kB
[ebuild  N    ] sys-libs/gpm-1.20.1-r5  559 kB
[ebuild  N    ] sys-libs/ncurses-5.5-r2  USE="gpm unicode -bootstrap -build -debug -doc -minimal -nocxx" 2,259 kB
[ebuild  N    ] app-shells/bash-3.1_p16  USE="nls -afs -bashlogger -build" 2,514 kB
[ebuild  N    ] sys-libs/readline-5.1_p4  1,986 kB
[ebuild  N    ] virtual/libiconv-0  0 kB
[ebuild  N    ] sys-devel/gettext-0.14.5  USE="emacs nls -doc -nocxx" 6,939 kB
[ebuild  N    ] sys-apps/diffutils-2.8.7-r1  USE="nls -static" 1,037 kB

and finally, in line 75:

Code: Select all

[ebuild  N    ] sys-libs/glibc-2.4-r3  USE="nls nptl nptlonly -build -glibc-omitfp -hardened -profile" 0 kB 

That's why. The order is not optimal. glibc should not be the 75th package to be compiled, but one of the first ones.

Basically, my script does nothing else than run an emerge -e world, only that it reorders items, and excludes GCC from the list, because GCC has already been installed.

"Is 'emerge -e' flawed then?" one may ask.

No. But "emerge -e" assumes there is absolutely nothing there when it starts its job.

Which is not the reality if you are running an emerge from an installed gentoo System.

For instance, you may already have the new GCC, binutils etc. installed.

Exploiting this fact, you can optimize the order in which "emerge -e" does things, and even skip some of the packages (GCC).

Besides that, my script is identical to "emerge -e" world.

It does nothing what you couldn't do manually. (In fact, I did all the steps manually before writing the script.)

So my script is only by one packet faster than what "emerge -e" does.

However - "emerge -e" has not yet done the full job!

Remember those 75 packages that have been compiled before glibc was updated?

"Just to be sure" one should recompile those 75 packages again after the emerge finishes, so that all of them are linked against the new glibc (although I doubt this will be necessary... but that's another story).

My script compiles glibc as the first package (emerges it as the second package; the first package is only header files), and thus makes any this sort of headache obsolete from the beginning.

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

Well, that's the problem.

Actually, it is not even necessary to recompile all packages which have been compiled with the old compiler, because the C ABI is pretty stable (see my posting with the explanations for my guide for more of this).

It would be sufficient to recompile only those source files, which have been written in the C++ language by the old compiler.

Unfortunately, it's hard if not even possible to find that out in a safe, automated way (see also my explanation posting).

So we better recompile everything, just to be sure not to miss one of those C++ sources.

Plus we have the benefit that then everything is compiled using the presumably better code generator of the new compiler.

[grimm26]

re system profile and glibc. So, I should recompile glibc after setting my system profile? Also, when I did an eselect profile list, it didn't show one as selected. It also did not show 2006.1/server as a choice, only desktop.

[dalek]

It doesn't show up here either but it is there. It is here: /usr/portage/profiles/default-linux/x86/2006.1/server In the same place as desktop no less.

What's up with that??

[Guenther Brunthaler]

re system profile and glibc. So, I should recompile glibc after setting my system profile? Also, when I did an eselect profile list, it didn't show one as selected. It also did not show 2006.1/server as a choice, only desktop.

Then your system seems not to be up to date, or something is screwed. When I run

Code: Select all

$ eselect profile list

I get

Code: Select all

Available profile symlink targets:
  [1]   default-linux/x86/2006.1
  [2]   default-linux/x86/no-nptl
  [3]   default-linux/x86/no-nptl/2.4
  [4]   default-linux/x86/2006.1/desktop
  [5]   hardened/x86
  [6]   hardened/x86/2.6

So you see it's profile # 1 on my box.

In order to be sure your system is really up to date, run the following

Code: Select all

emerge --sync && emerge --update --deep --newuse world && revdep-rebuild

It that won't help then I'm also clueless...

[Erlend]

I've always recompiled xorg from within xorg and I don't remember having any problems - I'm not saying you won't have problems just that any problems I had were so minor I never noticed.

That's good news!

But - what Desktop Environment are you using?

I suspect some environments to be more fragile than others when recompiling them while they are running.

I'm currently using Xfce. I previously used Gnome.

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

Well, that's the problem.

Actually I think it could be done if you ask the user when they changed their compiler to (say) gcc 4.1.1. Just look up emerge.log for any packages done after this date (although they would probably not be linked against the new glibc if you do this).

Plus we have the benefit that then everything is compiled using the presumably better code generator of the new compiler.

Off topic, but I can't wait until icc works as a drop in replacement for gcc (if ever)!

[Naib]

talk abt postcount++++++++

[Erlend]

Hey I just happened to spot your script re-emerging =sys-libs/libstdc++-v3-3.3.6 ... surely I don't need that program any more?

Thanks,

Erlend

[Guenther Brunthaler]

I'm currently using Xfce.

Hmm that's also one of the leaner ones. The odds of "surviving" a recompile without crashing are certainly quite better with such a DTE than for KDE!

Furthermore, can you tell whether Xfce is written in C++ or C?

The maximum damage is done only if C++ Shared Objects are heavily used (as plugin DLLs).

I also assume Gnome to be much less effected by a recompile than KDE for that reason.

Actually I think it could be done if you ask the user when they changed their compiler to (say) gcc 4.1.1. Just look up emerge.log for any packages done after this date (although they would probably not be linked against the new glibc if you do this).

Yes, that might actually work.

And it can rather easily be included within my guide: After the recompilation script has been generated, remove any lines from it which refer to packages found in the output of emerge.log since the new compiler was emerged and was made the system default compiler. (Because otherwise it will not have been used although it was installed.)

A small Perl/Python/Ruby/awk script will do.

Unfortunately, it will only work for people who have enabled portage logging - it's still not the default, I'm afraid. (Is that correct?)

Off topic, but I can't wait until icc works as a drop in replacement for gcc (if ever)!

Arrrgh! A non-free compiler... <paranoid>Will I ever trust code generated by such a compiler? Who knows what it embeds into my object code? 50 % Intel code, 50 % NSA code? Shiver...</paranoid>

Don't mind!

[Lloeki]

it took me 2 days, but my kde laptop survived while using it extensively at work. I even hibernated 2 times.

sure kde is plenty of c++, but once a shared lib is loaded, it stays in mem until unused, so you can safely use your computer while emerging world. just be sure not to close c++ programs you wish to use (and start them beforehand of course). then, at the end of the upgrade, restart X, or better even, reboot, so the previous libs are flushed and latest libs reloaded.

I basically didn't noticed it up until I accidentally closed kmail, which then complained about the ABI mismatch. but who cares, I waited next day to check my mail again. just a fine, hassle-free upgrade.

[Guenther Brunthaler]

just be sure not to close c++ programs you wish to use (and start them beforehand of course). then, at the end of the upgrade, restart X, or better even, reboot, so the previous libs are flushed and latest libs reloaded.

This sounds very promising, indeed.

I have taken the liberty to update my guide accordingly and refer to your post!

Thanks a lot for sharing your experience with us!

[Lloeki]

you're welcome.

do you know about app-portage/genlop?
I didn't use your script, but I thought it would be a nice addition to your script to output the result of 'emerge -p `cat list_of_packages` | genlop -p' (i.e total estimated merge time) before actually running 'emerge `cat list_of_packages`', and have a yes/no question like 'emerge -a'.[/url]

[Guenther Brunthaler]

do you know about app-portage/genlop?

Well, all I was using it for until now was

Code: Select all

genlop -l | tail -30

and

Code: Select all

genlop -c

I skimmed quickly over the other options when first reading the manual, but didn't notice anything else which caught my immediate interest.

However, I'll give it another try now (after having read your comments).

i.e total estimated merge time

Actually, when writing my script I was thinking about some "ETA" display. But then I realized that date/time arithmetic is a bit outside the scope of a humble shell script, and so I let it be.

I was also uncertain how to best calculate that ETA, because my script is incremental and may be stopped and restarted at any time.

Futhermore, users may run different processes during the running time of my script, which makes it hard to give a good estimate.

But ultimately I was just too lazy to do it!

[vicaya]

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way: emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

[devsk]

I was also uncertain how to best calculate that ETA, because my script is incremental and may be stopped and restarted at any time.

you need to pipe the output of emerge -p (or equivalent generated by your script) to genlop -pt. Try this:

Code: Select all

emerge -pe system|genlop -pt

[Guenther Brunthaler]

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way

Without question, ccache is a great tool. I'm using it all the time.

And it works perfectly with my script. (At least I did not encounter any problems. However, I manually cleared the cache of ccache after using gcc-config to set the new compiler. Just to be sure there are no cached object files left originating from the old compiler.)

emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

I that only were so easy ... I would never needed to write my script.

If you just do a "emerge -e world", you will eventually find yourself with a recompiled system where about 75 packages have been linked against the old version of glibc ... see http://forums.gentoo.org/viewtopic.php? ... 09#3552919 for more of this.

So, if you want to be "100 % sure", "emerge -e world" won't provide you with what you want.

My script does the same as an "emerge -e world", only in the correct order for recompiling a system.

Also note that dispite of this, "emerge -e world" is neither flawed nor broken! It's just not intended to be used for recompiling the whole system, but for rebuilding a non-existent system from scratch. Which is not the same!

My script, in contrary, has been designed for recompiling your entire system.

So it does a similar, but not the same job.

[Erlend]

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way: emerge -e system and then world.

ccache won't help in this instance because the compiler version has changed.

It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

You won't be 100% sure, because as stated above 70-or-so packages won't be linked against the newly compiled glibc... this is why some people say you should do

Code: Select all

emerge -e system
emerge -e world
emerge -e world

The optimisation this script makes is to place glibc at the beginning of the compile run, so everything thereafter is compiled against it and running "emerge -e world" twice is definitely unnecessary.

Yes, to the environment! Thousands and thousands of Gentoo users, all wasting weeks worth of processing power when following inefficient "recompile entire system"-style compiler upgrade guides, will waste an enormous amount of kilowatt hours of electrical power.

How much more electricity does it use? The processor will go to 100%, and I guess the hard drives will thrash? Hard drives are always spinning though, so I don't think they'll really use significantly more power whem read/writing - it'll just kill the disks. The fans will probably be going quite fast too?

[Erlend]

Also note that dispite of this, "emerge -e world" is neither flawed nor broken! It's just not indended to be used for recompiling the whole system, but for rebuilding a non-existent system from scratch. Which is not the same!

There should be some way of doing what your script does from within portage... like emerge --rebuild-upgrade world.

[Guenther Brunthaler]

Hey I just happened to spot your script re-emerging =sys-libs/libstdc++-v3-3.3.6 ... surely I don't need that program any more?

Well, my script recompiles any package that is installed at the time it is generated. It doesn't care about whether the packages are up-to-date or outdated. (That's why I want users to run "emerge --update --newuse --deep world" first, before starting my script.)

So it seems your package is simply left from an old compiler and never has been uninstalled.

OK, now it is too late... but as soon as all is finished, run a

emerge --ask --depclean

and see what portage suggests for uninstallation. (And don't forget to run revdep-rebuild in order to fix any packages which might have been broken by those uninstalls.)

Another common problem are old slot installations.

For instance, GCC is not replaced when you emerge a newer version, but installed additionally.

Same for gentoo-sources, by the way.

If you want to see what slots of a package are actually installed, try

Code: Select all

equery list gcc
equery list gentoo-sources

Then use "emerge --unmerge" to uninstall the slot versions you do not need. But be sure to specify the exact version number for the unmerge, or else the newest version will be removed instead!

[Erlend]

Hey I just happened to spot your script re-emerging =sys-libs/libstdc++-v3-3.3.6 ... surely I don't need that program any more?

Well, my script recompiles any package that is installed at the time it is generated. It doesn't care about whether the packages are up-to-date or outdated. (That's why I want users to run "emerge --update --newuse --deep world" first, before starting my script.)

So it seems your package is simply left from an old compiler and never has been uninstalled.

Sorry I meant in general... I don't think I need libstdc++v3 any more - I think I emerged it to fix any problems that might occur in the 3.5 -> 3.6 upgrade.

[Guenther Brunthaler]

You won't be 100% sure, because as stated above 70-or-so packages won't be linked against the newly compiled glibc...

Yes!

How much more electricity does it use?

It certainly depends on the processor and its power-saving capabilities.

And also on the OS.

Unless disabled, Linux will perform a HLT instruction on the processor when no processes need running.

Then the processor "sleeps" until the next interrupt, requiring only very little power.

But when running under full load, such as recompiling QT libraries - I'm quite sure the processor needs much more power then, in comparison to "sleeping". (You can even hear it work when listening to the variable fan noise.)

And also don't forget: Some people turn off their machines at night! But for the updates they have to let them running for a week or so. Certainly this also uses more power than usual.

and I guess the hard drives will thrash?

At least they will if those horror-stories from GCC 4 are actually true - I've read reports where the machines were running out of swap space, because GCC4 needed more than 2 gigs to compile a single C++ file.

I can only hope those guys were exaggerating a bit... because one of my machines has only 512 megs.

But for now I'll stay with 2006.0 for a while - I have updated my GCC from 3.3 to 3.4 not more than two months ago - it's no fun recomiling everything again so soon ;-(

Hard drives are always spinning though

Yes, but there are 2 motors in a hard drive: The normal one
which makes it spin, and the stepper motor. At least in older hard disks there was one. Newer hard disk have something else (I think its called "actuator" or similarly), but it certainly needs also more power if its moving the heads, than if not moving them.

The fans will probably be going quite fast too?

You can bet on that! Especially on my notebook here - "you can hear the sound"...

[Erlend]

At least they will if those horror-stories from GCC 4 are actually true - I've read reports where the machines were running out of swap space, because GCC4 needed more than 2 gigs to compile a single C++ file.

Ah that explains it... my swapfile is going through the roof (and I can only remember 3-4 occassions when it's been used before).

But it's not just the swapping that'll use disk space, there's also the fast creation of hundreds of small files in /var/tmp, although I think PORTAGE_TMPFS="/dev/shm" in make.conf is meant to lower this.