Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Tue Sep 05, 2006 11:31 pm    Post subject: Re: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ Reply with quote

outspoken wrote:
Your guide should read more like this:


OK, so you are suggesting to do the gcc-config before the emerge --update then.

It should certainly not a be problem changing this in my guide, but first I have to understand what exactly the problem was.

From what I've seen in the link you provided, they were actually getting a "gcc too old"-sort of error message.

But in our case, GCC will be up to date when the glibc will finally be compiled by my script, so I am uncertain on what step of my guide the problem actually occurred in your case!

So far, my the logic of my guide is:

  • The initial emerge --update brings all packages up-to-date, including glibc and the old GCC as well as all the tools required for rebuilding the system.
  • Then the new GCC is emerged. Now both GCCs are installed, and still all packages are up-to-date.
  • Now the used runs gcc-config and sets the new GCC as the system default compiler. But the user does not use it to compile anything with it yet.
  • Instead the user starts my script, which will generate a package list of all packages in the system, and also determine the correct order in which they should be recompiled. This includes glibc.
  • The user starts the generated script. glibc should be emerged as the second package of all, and the first one to be actually compiled. This will already be done with the new GCC, which is the system C compiler now.


If there is an error in this flow of logic, please be more specific and tell me at what point of the above list the problem occurs.

I may then be able to realize the origin of the problem. (Let's hope!)
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Tue Sep 05, 2006 11:45 pm    Post subject: Reply with quote

Well, I did mine the hard way. I didn't see this guide before I started. I have ran revdep-rebuild about 3 times now. It wants to rebuild gcc each time and it just goes in circles. Do you have any reason why it does this?? Info:

Code:
[I--] [  ] app-portage/portage-utils-0.1.20 (0)


Which I seem to recall has revdep-rebuild in it.

Code:
root@smoker / # emerge --info
Portage 2.1-r2 (default-linux/x86/2006.1/desktop, gcc-4.1.1, glibc-2.4-r3, 2.6.16-gentoo-r7 i686)
=================================================================
System uname: 2.6.16-gentoo-r7 i686 AMD Athlon(tm) XP 2500+
Gentoo Base System version 1.12.4
ccache version 2.3 [enabled]
app-admin/eselect-compiler: [Not Present]
dev-lang/python:     2.3.4-r1, 2.4.3-r1
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.3
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.59-r7
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.13-r3
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=athlon-xp -O2 -fomit-frame-pointer -momit-leaf-frame-pointer -fno-ident -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo"
CXXFLAGS="-march=athlon-xp -O2 -fomit-frame-pointer -momit-leaf-frame-pointer -fno-ident -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig buildpkg ccache distlocks metadata-transfer parallel-fetch sandbox sfperms strict"
GENTOO_MIRRORS=  < snip >
LDFLAGS="-Wl,-z,now"
LINGUAS="en"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 3dnow X acl acpi alsa amd arts artswrappersuid avi berkdb bitmap-fonts bzip2 cairo cdr chroot cli crypt cups dbus dlloader doc dri dvd dvdr eds emboss encode esd exif fam fdftk fortran gaim gcj gdbm gif gimp gimpprint gkrellm gphoto2 gpm gstreamer gtk hal hbci ipv6 isdnlog java javascript jbig jpeg jpeg2k justify kde ldap libg++ mad mikmod mmx mp3 mpeg ncurses nls nptl nptlonly nsplugin offensive ofx ogg opengl pam pcre pdflib perl png postgres ppds pppd python qt3 qt4 quicktime readline reflection samba scanner sdl seamonkey session spell spl sqlite sse ssl tcltk tcpd tiff tk truetype truetype-fonts type1-fonts udev unicode usb vorbis win32codecs wmf xml xmms xorg xprint xv yahoo zlib elibc_glibc input_devices_keyboard input_devices_mouse kernel_linux linguas_en userland_GNU video_cards_nvidia"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

root@smoker / #               


I ran into the same thing with the old compiler before the upgrade. It just wants to do both now like this:

Code:
root@smoker / # revdep-rebuild -p
Configuring search environment for revdep-rebuild

Checking reverse dependencies...

Packages containing binaries and libraries broken by a package update
will be emerged.

Collecting system binaries and libraries... done.
  (/root/.revdep-rebuild.1_files)

Collecting complete LD_LIBRARY_PATH... done.
  (/root/.revdep-rebuild.2_ldpath)

Checking dynamic linking consistency...
  broken /usr/lib/aqbanking/plugins/0/bankinfo/de.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/aqbanking/plugins/0/bankinfo/de.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/aqbanking/plugins/0/imexporters/ofx.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/aqbanking/plugins/0/imexporters/ofx.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/aqbanking/plugins/0/providers/aqofxconnect.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/aqbanking/plugins/0/providers/aqofxconnect.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/avifile-0.7/ac3pass.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/ac3pass.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/audiodec.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/audiodec.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/ffmpeg.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/ffmpeg.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/mad_audiodec.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/mad_audiodec.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/mp3lame_audioenc.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/mp3lame_audioenc.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/mp3lamebin_audioenc.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/mp3lamebin_audioenc.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/mpeg_audiodec.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/mpeg_audiodec.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/osmjpeg.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/osmjpeg.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/vorbis_audio.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/vorbis_audio.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/win32.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/win32.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/avifile-0.7/xvid4.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/avifile-0.7/xvid4.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/lib-org-w3c-dom.la (requires /usr/lib/libgcj.la)
  broken /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/lib-org-xml-sax.la (requires /usr/lib/libgcj.la)
  broken /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libgcjawt.la (requires /usr/lib/lib-gnu-java-awt-peer-gtk.la)
  broken /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libgij.la (requires /usr/lib/libgcj.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkio.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkdesu.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkwalletclient.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkdeui.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkdecore.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libDCOP.la)
  broken /usr/lib/kde3/libk3blibsndfiledecoder.la (requires /usr/kde/3.4/lib/libkdefx.la)
  broken /usr/lib/libaqbankingpp.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/libaqbankingpp.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/libaqofxconnect.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/libaqofxconnect.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/libaviplay.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/libaviplay.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/libkbanking.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/libkbanking.la (requires /usr/lib/libgwenhywfar.la)
  broken /usr/lib/libqavm.la (requires /usr/lib/libaviplayavformat.la)
  broken /usr/lib/libqavm.la (requires /usr/lib/libaviplayavcodec.la)
  broken /usr/lib/libqbanking.la (requires /usr/lib/libaqbanking.la)
  broken /usr/lib/libqbanking.la (requires /usr/lib/libgwenhywfar.la)
 done.
  (/root/.revdep-rebuild.3_rebuild)

Assigning files to ebuilds... done.
  (/root/.revdep-rebuild.4_ebuilds)

Evaluating package order... done.
  (/root/.revdep-rebuild.5_order)

All prepared. Starting rebuild...
emerge --oneshot -p =sys-devel/gcc-3.4.6-r1 =sys-devel/gcc-4.1.1

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] sys-devel/gcc-3.4.6-r1
[ebuild   R   ] sys-devel/gcc-4.1.1
Now you can remove -p (or --pretend) from arguments and re-run revdep-rebuild.
root@smoker / #   


You have any ideas why it sends me in circles?

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Tue Sep 05, 2006 11:45 pm    Post subject: Reply with quote

devsk wrote:
I think it uses ldd to determine if all the executables/libraries have there dependent libs present. And that's it.


Which meant I needed to write a script that mimics revdep-rebuild's job, only that is also uses nm on every SO in order to verify all of its entry points.

In order to be able to do that, I need to do a second pass over all SOs and executables, collecting all references to other SOs.

And all that for only a very few cases where this is actually needed.

Hmmm. Sounds not exactly fun to implement... Perhaps, if I should be very bored at some time... ;-)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Wed Sep 06, 2006 12:09 am    Post subject: Reply with quote

dalek wrote:
You have any ideas why it sends me in circles?


Hard to say, as it certainly depends on the set of packages you have installed.

But I can acknowledge the problem per se, although I experienced it with some different package. (Don't remember which one.)

However, I still remember that I found the origin of the problem: It was the consequence of a mixture of normal Gentoo packages installed from source code, and prebuilt binary packages.

It was either mozilla-firefox-bin, mozilla-thunderbird-bin or openoffice-bin. Don't remember which one.

After uninstalling the guilty bin package, the circles were gone.

I think the problem were the USE flags.

The faulty bin-package was compiled without some optional feature, which was required because of USE-flag settings by another package.

Because of this missing feature, some library entry point in some shared library from the bin-package was undefined, although the other package required it (as a consequence of the USE-flags).

revdep-rebuild detected the missing library-entry, and thus scheduled the faulty bin-package for re-emerging.

But as re-emerging does not change anything for a bin-package, the problem continued to exist after the re-emerge, and so the next run of revdep-rebuild detected the problem again, emerging the package again... and again... and again...

You get the picture.
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Wed Sep 06, 2006 3:02 am    Post subject: Reply with quote

Thanks for the info. As I said I ran into this before and I usually let it do it twice then I figure the point is pretty much mute. I don't think I have to much binary stuff installed on here though. I even compile OOo on this thing. I may start a seperate thread, as much as I hate to, and see if it is a bug that needs to be dealt with before it does break something for someone else.

I did get your script and I stopped the emerge -e world that I had running. I then ran it and it seems to be working fine. I noticed that it compiles the kernel stuff first and that it said to compile glibc again during the install. I thought hmmm, never saw that before. Then it started emergeing glibc just like it said too. That was sort of funny in a way. It's like the script was reading the things mind. 8O

Anyway, I'll try revdep-rebuild again after your script gets done. If I get the same thing I'll start a thread.

Thanks

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
vicaya
n00b
n00b


Joined: 26 Jun 2004
Posts: 57

PostPosted: Wed Sep 06, 2006 9:15 am    Post subject: Reply with quote

Guenther Brunthaler wrote:
vicaya wrote:
emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

I that only were so easy ... I would never needed to write my script.

Sorry, what I really meant is that just follow the official gcc upgrade instruction. emerge gcc etc... see http://www.gentoo.org/doc/en/gcc-upgrading.xml code listing 2.1 and then emerge glibc && emerge -e system && emerge -e world. With ccache, it won't be that much slower (considering the total emerging time) than any "optimal" script.
Back to top
View user's profile Send private message
Bad Penguin
Guru
Guru


Joined: 18 Aug 2004
Posts: 507

PostPosted: Thu Sep 07, 2006 12:02 am    Post subject: Re: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ Reply with quote

Guenther Brunthaler wrote:

(can a Penguin be bad at all? Not for me. I like penguins!)


What, you didn't know that linux users are thieves, liars, terrorists, and long haired smellies? Where have you been ;)

The Bad Penguin is a barb at the litigious assholes known as the SCO Group, in particular Darl McBride, who has falsely accused the entire open-source community of being thieves...

Guenther Brunthaler wrote:

But you are not the first one to find a flaw in my guide. Hey, I'm only human! I'm not perfect, everyone can make as mistake and I'm no exception. That's why we discuss ideas like this in the forums, don't we?

For instance, devsk has already pinned down a potential problem related to circular dependencies, although it seems that (by coincidence I must admit) my script will not be affected by it. See his posting for more.

The flaw is not so much in the guide, but in making definitive, authoritative sounding statements that have not been tested or proven. I am just anal that way ;)

Guenther Brunthaler wrote:
I have not even an idea how to prove the correctness of a computer program with as far-reaching consequences as the application of my script. Perhaps if I wrote it in ADA instead as a BASH script... ;-)

Perhaps by building it via your method and running the test suites of every package as you go (FEATURES="test")? The only way to be certain is to actually use your method in some different environments and see how many bugs are produced as a result.

Guenther Brunthaler wrote:
Bad Penguin wrote:
As far as your second assertion, that your method is somehow faster, wrong again.


Here we obviously have a simple misunderstanding: When I wrote
Quote:
Compared to other guides which suggest recompiling the entire system up to 6 times
I didn't mean I my guide was faster than all the guides which may exist, only faster than those which recommend excessive recompilation of the system over and over.

Which, by the way, was the primary motivation for writing my guide: 6 times was too much!

Heh heh, I have never seen a guide that suggests to do it 6 times. The Rob Moss method suggests to "emerge -e system && emerge -e system && emerge -e world && emerge -e world", which compiles the system packages four times and the remaining packages (world) twice. This method is not very useful for toolchain updates though, which need to be done in the correct order.

Guenther Brunthaler wrote:
As you have read my argumentation and how my script works, you might have noticed that it builds a list of all packages to be recompiled when the generated build script is run.

This list countains all the packages which will be rebuilt in the correct order, including any dependencies.

An incorrect assumption. If you are merely recompiling every package on your box for fun, or a change in USE flags/CFLAGS, you would be correct. The only reason to ever recompile every package would be a toolchain update, which needs to be done in a specific order, which is not handled by portage correctly.

Guenther Brunthaler wrote:
But an update like the 2006.1 switch is mere overkill for the power of your script, where my simple guide is obviously sufficient for such minor cases.


That depends on which profile and versions of everything you are upgrading from. Not everyone is going to be switching to the 2006.1 profile from a 2006.0 profile. Your script might work in a very small set of cases, in other cases it might turn out to be disastrous. You are leaving one very important factor out of your method - the human factor ;) Bugs happen, people don't always know to go to the gcc/glibc migration guide, which gives out bad advice anyway. Use revdep-rebuild on your system and you will soon find out that you have old .la files laying around that will not be removed by emerge because the files have been modified by an outside process. Who the hell knows what code is actually being statically compiled into your packages as a result.

Guenther Brunthaler wrote:
But why should such a compiler actually be faster than the compiler emerged at the beginning of my guide, which fully honors all MMX/SSE flags and also uses the full optimization settings as defined in /etc/make.conf?

And from then on, my compiler has to recompile all remaining packages in the system exactly once, according to my guide.

During a bootstrap the USE flags are only restricted during the bootstrap phase, when the actual toolchain packages are built. The subsequent emerge -e system uses any USE flags in make.conf. One example of why this is useful is the solitaire game pysol - which requires tk. If all use flags were allowed in the bootstrap phase the compile of python will pull in X to build tk, then you would have to build X all over again during the subsequent emerge -e system.

Guenther Brunthaler wrote:
How can your script, starting with a slower compiler, do the same in a faster way?

So your script/guide has to recompile at least exactly the same number of packages as my guide does.

It does not actually compile anything faster, it does it more efficiently. The important thing is not that your packages compile fast, it is that they compile correctly so you don't have to go back and do it all over again later. But no, my "method" actually only ends up compiling the bootstrap packages twice along with a small set of the bloated needs of portage (python). You start with a cleanly bootstrapped system, the remaining packages are all compiled on a clean toolchain.

Guenther Brunthaler wrote:
The "overhead" of my guide is the initial "update world", and the overhead of your guide is the overhead of a stage1 reinstall.

The "overhead" of your guide is that chances are it will really screw up peoples systems ;) Sadly, there is no way to guarantee that every package on the system is built against current libraries that are built against current libraries... yadda yadda yadda... without a brute force, recompile it multiple times approach. There are just too many circular dependencies and too many different factors when you take USE flags into account. The safest, and fastest way, for the large majority of people, is to start from stage1. This is unique to gentoo because it is a source based distribution... Too bad gentoo has abandoned the build from source approach, instead encouraging people to hack up their boxes... I digress.

Anyway, if you are interested, I will write up some scripts that take a box from a specific state, say portage-20060301, to portage-20060905. One script that does it your way, one that does it my way, and we can add FEATURES="test" to see how it turns out ;)
Back to top
View user's profile Send private message
Sculler
n00b
n00b


Joined: 09 Dec 2005
Posts: 16

PostPosted: Thu Sep 07, 2006 11:57 am    Post subject: Reply with quote

It might be a good idea to reorder the prerequisites to emerge gentoolkit is before revdep-rebuild, because (I think) that command is part of gentoolkit.
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Thu Sep 07, 2006 12:10 pm    Post subject: Reply with quote

Well, the script just finished. I have the same problem that I had before the gcc upgrade just with the newer version so I guess all is well. It seems revdep-rebuild just likes to emerge gcc in a circle.

Oh well. It all works at least. :roll:

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Thu Sep 07, 2006 1:47 pm    Post subject: Re: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ Reply with quote

Bad Penguin wrote:
Anyway, if you are interested, I will write up some scripts that take a box from a specific state, say portage-20060301, to portage-20060905. One script that does it your way, one that does it my way, and we can add FEATURES="test" to see how it turns out ;)


Interested, yes, of course.

But I'm afraid my knowledge about Portage and the internals of ebuilds is not yet mature enough to write any "test" scripts myself. Those internals are mostly still black magic for me.

In fact I have not even finished studying the autoconf guide - and I'm even farther from writing my own ebuilds.

After all, I'm relatively a Linux newbie and there are still many aspects of the system which I don't fully understand (or understand at all). Although I am trying hard to learn. But it takes time.

So just give me a little more time please ...
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Thu Sep 07, 2006 1:51 pm    Post subject: Reply with quote

Sculler wrote:
It might be a good idea to reorder the prerequisites to emerge gentoolkit is before revdep-rebuild, because (I think) that command is part of gentoolkit.


Thanks for pointing that out! I'll check to which package revdep-rebuild actually belongs and will reorder the prerequisites accordingly.
Back to top
View user's profile Send private message
ANewFishMonkey
n00b
n00b


Joined: 04 Sep 2006
Posts: 1

PostPosted: Fri Sep 08, 2006 10:02 am    Post subject: Reply with quote

I have read through this thread, I think it will solve my problem but I’m not entirely sure,.
I have recently migrated to X Server 7.0, which I sort of got working, (but so much is broken its not useful!).
Some of the new X packages (glibc I think) failed to compile, due to an insistence of using gcc 3.4.6 not 3.3.6, so I switched and tried again. The problem was when I recompiled my kernel, all of the modules refused to load, complaining of “magic version” different. (the complier version namely), but I did recompile all the modules with the same complier as the kernel. (or at least I think I did )
What I want to know is :
Is there some other setting I need to change other than gcc-config to ensure modules and kernel are compiled using the same version?

Does 2.6.12-gentoo-r4 HAVE to be compiled with 3.3.6?

I “got round” this issue, by compiling the kernel and modules with 3.3.6 everything else with 3.4.6, but now I have messed up because the ATI fglrx module complains about missing symbols. I pretty sure your script is what I need, but I need to know what else I need to do first, not mad about wasting electricity and all that.
:D

BTW, I'm currently using 2006.0 profile, shall I switch to 2006.1 and just use gcc 4.1 saving time later?
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Fri Sep 08, 2006 5:51 pm    Post subject: Reply with quote

ANewFishMonkey wrote:
Is there some other setting I need to change other than gcc-config to ensure modules and kernel are compiled using the same version?


No, that should be sufficient.

However, you also need to recompile all the packages compiled with the old compiler, because mixing packages compiled with different major compiler versions can cause troubles.

Unfortunately, portage is not aware about those problems, and will not automatically recommend recompiling all the packages. You have to do this on your own.

Common sense is that all packages should be recompiled at least once, which is what my script does.

There are different points of view whether this is enough; I say "yes". (See my argumentation link in the article.)

Recent major GCC updates which require recompiling the system are: From pre-3.4 to 3.4, and from pre-4.0 to 4.x.

As you are updating from 3.3.6 to 3.4.6 or 4.1 you are affected anyway.

BTW: Just switching back to 3.3.6 won't help if you already compiled some packages with 3.4.6: Either you recompile all those packages again with 3.3.6, or you will end up in a system with mixed versions, which will cause troubles sooner or later for sure.

I thus suggest completely updating your system to either 3.4.6 or 4.1 to have a stable system again. You may use my script for this, or some of the other guides (which take a longer time, but their authors believe it's better. I don't believe this, but you decide! The most pragmatic version is probably using my guide first, because it's fastest under normal circumstances, and if you should be so unlucky to be the first one having problems with it, then you still can use one of the other guides.)

ANewFishMonkey wrote:
Does 2.6.12-gentoo-r4 HAVE to be compiled with 3.3.6?


No. The only known limitation I am aware of is that GCC4 cannot be used to compile a kernel 2.4 or older.

3.3.6 as well as 3.4.6 or 4.1 should not have any problems compiling a 2.5+ kernel.

ANewFishMonkey wrote:
ATI fglrx module complains about missing symbols.


I also had that problem, although with the NVIDIA drivers.

It was not a problem of the compiler, but of the kernel version: They had changed something in the new kernel version, which also required a corresponding change in the NVIDIA code.

When an update was available to the NVIDIA binary drivers, all the problems went away.

In meantime, I was using the open source drivers instead. Although they had inferior 3D speed (= unusable for games), there was no difference for normal X applications.

So I did not play 3D games for a couple of weeks, but otherwise there was no disadvantage using the open source X11 drivers until the new NVIDIA binary drivers were finally released.

I'm pretty sure things will be the same for ATI.

ANewFishMonkey wrote:
BTW, I'm currently using 2006.0 profile, shall I switch to 2006.1 and just use gcc 4.1 saving time later?


Yes, you can. I did it also. Just look into the official Gentoo GCC update guide (but don't follow it if you want to use my guide). They have a paragraph there descibing how to mask GCC4 in the 2006.1 profile, so that GCC 3.4.6 will be used instead.

But - is there a specific reason why you won't be using GCC4? If you recompile your entire system anyway, why not using the best compiler if you can, which certainly is 4.x?

Sure, 4.x works slower than 3.x, which means it takes more time to recompile your system.

But then, 4.x also generates better code, which means the programs it compiles will run (slightly) faster then. GCC4 is slower because it does a better job, which takes more time to complete.

The only problem of GCC4 is its hunger for RAM. If you have a box with less than 512 megs, it may be better to stay with 3.4.6.

But otherwise, I would suggest using 4.x.


Last edited by Guenther Brunthaler on Sat Sep 09, 2006 10:36 pm; edited 1 time in total
Back to top
View user's profile Send private message
Lloeki
Guru
Guru


Joined: 14 Jun 2006
Posts: 437
Location: France

PostPosted: Fri Sep 08, 2006 6:15 pm    Post subject: Reply with quote

Code:
$ free
             total       used       free     shared    buffers     cached
Mem:       1554960    1244864     310096          0     137388     872464
-/+ buffers/cache:     235012    1319948
Swap:      3502128      71036    3431092

While compiling with kde, firefox & al running, I wouldn't call that exactly hunger. it would fit in 256M. (btw, currently compiling OOo203)
_________________
Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F*** ;)
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Fri Sep 08, 2006 7:35 pm    Post subject: Reply with quote

Lloeki wrote:
I wouldn't call that exactly hunger. it would fit in 256M. (btw, currently compiling OOo203)


Looks good; yes.

However, this is a snapshot from one instance in time. The current source file obviously does not need much, but who can say the next source file will also have the same decent requirements.

In order to analyze the memory requirements, it would be necessary to collect the output in a loop, taking a sample all couple of seconds and write it to a log file.

Afterwards, we can then search for the sample with the maximum and minimum memory requirements, and also calculate the average requirements.

A loop similar to

while true; do free; sleep 10; done > memory.log &

should do the job (it must be manually killed when the recompilation is done).

Of course, now it may be too late, if you're almost done recompiling. But perhaps the next time.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Fri Sep 08, 2006 7:47 pm    Post subject: Reply with quote

Lloeki wrote:
Code:
$ free


I just looked at the man page for free; perhaps it's not necessary to run free in a loop and analyze the log contents afterwards.

free has a -l option which reports low and high memory usages, and the -s <N> option repeats the output all <N> seconds.

However, I don't know whether the -l values are cummulative, or just per-sample. That needs further investigation...
Back to top
View user's profile Send private message
mirek
Guru
Guru


Joined: 20 Sep 2004
Posts: 489
Location: Oslo Norway

PostPosted: Fri Sep 08, 2006 11:41 pm    Post subject: Reply with quote

I have got an error
Code:
# sh genscript.sh
Decoding attachment - please standby.
genscript.sh: line 7: syntax error near unexpected token `||'
genscript.sh: line 7: `|| die "I need openssl for base64 decoding!" '


when I run sh genscript.sh script
I have openssl installed
Code:
# emerge -p openssl

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] dev-libs/openssl-0.9.8c

and test for it
Code:
# echo 'dGVzdGluZwo=' | openssl base64 -d
testing


Last edited by mirek on Sat Sep 09, 2006 8:28 am; edited 1 time in total
Back to top
View user's profile Send private message
blackcell
n00b
n00b


Joined: 17 Aug 2002
Posts: 56
Location: Oregon

PostPosted: Sat Sep 09, 2006 2:14 am    Post subject: Reply with quote

Great Guide and script :D

I had no problems first time through all the way to completion, 40 hours later. Only failure was mod_gzip fails to compile.
Gzip bug
_________________
"If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year, killing everyone inside."
Back to top
View user's profile Send private message
agrypa1
Apprentice
Apprentice


Joined: 31 Mar 2005
Posts: 244

PostPosted: Sat Sep 09, 2006 4:59 am    Post subject: Re: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ Reply with quote

Hi, thank you for the script. My system is being compiled right now, as we speak :-). I have an issue though which I resolved in two ways. I am not sure the first solution hasn't spoiled my system, though.
Namely, during the compilation process the script stopped with an error at a package from an overlay tree. No thinking much I unmerged the package and ran the script again. Of course it didn't work that way, because the package was still on the list of the script. So I removed the files the script generated (script.ar, remaining-packages, entire -system). I recreated the script and started anew. At this point I do not know whether gcc and linux headers were compiled again or not?

the script failed again at qemu-user and qemu-softmmu. This time I thought it would better be that I find a way to skip the compilation of the packages instead of removing them from my ssystem ans start from the beginning. So I found out that I can comment out the troublesome packages in the recompile-remaining-packages script. It worked.

Is this the only and correct way of recovering form a failure of a uncompilable package by commenting it out?

Thank you
Agryppa
_________________
The first successor of Saint Peter was Linus (a.d. 68-79) - whose namesake became the creator of Linux in our time. Torvalds' middle name is Benedict - the name assumed by the previous Pope who resigned from office.
Back to top
View user's profile Send private message
s_wilk
n00b
n00b


Joined: 22 Aug 2006
Posts: 27
Location: Lublin,PL

PostPosted: Sat Sep 09, 2006 11:12 am    Post subject: Reply with quote

Hi

Guenther Brunthaler wrote:
Quote:
However, if someone reading this has some bandwith left to waste, he/she is invited to host the decoded script at their web site and send me the download URL


So I put the script and a copy of the guide on my website (perserving information about the author).

You can find it here:
http://knip.pol.lublin.pl/~sw/gentoo/


BTW. I have a question to Guenther:

If I understand correctly Your script recompiles the packages in the exact versions as installed at the moment whith the same use flags (cause you say to do #emerge --ask --update --deep --newuse world before running your script)

So I assume I shouldn't use this script to switch to hardened profile from 2006.0?

Do you think it does make sense to do emerge -avu --deep --newuse world and then run your script?
I am not sure how much packages are affected with use flags change (hardened impicates at least -nptl hardened flags)
Back to top
View user's profile Send private message
juantxorena
Apprentice
Apprentice


Joined: 19 Mar 2006
Posts: 201
Location: The Shire

PostPosted: Sat Sep 09, 2006 2:31 pm    Post subject: Reply with quote

First of all, I don't want to offend anybody. I haven't tried this script, so I will say my opinions a priori:

1.- If this script works better than the emerge -e system && emerge -e world stuff that we can see in the official guide, why don't try to contact a developer to talk about this? Maybe the script can become an official part of Gentoo in the future. Have you contact any developer? What he/they say about this? Messing with the system like this is a dangerous thing, I don't want to break anything.

2.- Don't mean to offend you, but reading you post, the way they are writen and the way you explain things, I think that you are acting like a televangelist or an infomercial vendor. This undermines you credibility. If this thing works, talk with some devs in a chat, or something. But trying to "sell your product" with "Because the amazing truth is: It is in fact UNNECESSARY to recompile any package bla bla bla" or other similar quotes is not the best idea.

Personally I would wait to see what devs think about this.
_________________
I cannot write English very well. Please, correct any mistake so that I can improve.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Sat Sep 09, 2006 3:00 pm    Post subject: Reply with quote

mirek wrote:
I have got an error


I have re-checked it on my box; it still runs fine for me either using bash or ash.

It seems most likely to me there went something wrong with copy/paste.

Please verify the MD5 checksum:

Code:
cat > genscript.sh.md5 << END
07d0667e0893d99d4ddca0be0f2c2449 *genscript.sh
END
md5sum -c genscript.sh.md5
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Sat Sep 09, 2006 3:26 pm    Post subject: Re: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ Reply with quote

agrypa1 wrote:
No thinking much I unmerged ... So I removed the files the script generated (script.ar, remaining-packages, entire -system). I recreated the script and started anew.


You did well. This should fix any potential problems.

agrypa1 wrote:
At this point I do not know whether gcc and linux headers were compiled again or not?


"linux-headers" is really just that - a bunch of header files. They will only be downloaded and installed, but nothing will ever be compiled by emerging that package.

Besides that, re-emerging "linux-headers" is the very first thing my script does - so you certainly need not be worried about it.

GCC is another story: It won't be re-emerged by my script, because it is not necessary: If you were following my guide, you did this yourself already.

And as you can see in my related argumentation article ("myth 1 - GCC gets better each time it is re-emerged"), there is no point in re-emerging the same version of GCC ever again. It cannot get any better than it already is, no matter which profile or glibc was current at the time it was emerged.

So you also need not worry about re-emerging GCC too.

(The only case when re-emerging the same version of GCC makes sense is if the CFLAGS in /etc/make.conf or the USE flags for GCC have been changed by you.)

agrypa1 wrote:
Is this the only and correct way of recovering form a failure of a uncompilable package by commenting it out?


No, the lines could be deleted as well. But I preferred instructing the users of my guide to comment the lines out, because that way it's easy to un-comment the lines again in case they did comment out the wrong lines mistakenly.

Also, just commenting out allows looking at the script file contents after the system rebuild is complete, and see which packages were skipped.

Then the users could decide to try re-emerging those missing packages again, hoping it will then work as anything else will already be up-to-date.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Sat Sep 09, 2006 4:05 pm    Post subject: Reply with quote

s_wilk wrote:
So I put the script and a copy of the guide on my website (perserving information about the author).

You can find it here:
http://knip.pol.lublin.pl/~sw/gentoo/


Thanks very much for doing this!

I'll be to happy to update my guide and include your link.

s_wilk wrote:
If I understand correctly Your script recompiles the packages in the exact versions as installed at the moment whith the same use flags


That's correct. I wanted to use a stable version snapshot at some instance in time, or otherwise an imprudent "emerge --sync" could easily make an ongoing system rebuild operation break.

Using a stable version snapshot of all packages, the chances for an intermittent "emerge --sync" to break things are much lower.

s_wilk wrote:
So I assume I shouldn't use this script to switch to hardened profile from 2006.0?


There should not be a problem if you switch the profile before starting my script, because then the version snapshot will be stable again.

My script generates the package list based on the current system profile at the time the generator script is called.

And as my script will recompile everything anyway, all packages will be compiled according to the settings of the new profile.

With one exception: My script will never recompile GCC, because this will already have been done manually when users were following my guide.

This assumes however, that GCC has been emerged with the correct USE-flags and CFLAGS already set in /etc/make.conf (i. e. after the profile has been changed.)

s_wilk wrote:
Do you think it does make sense to do emerge -avu --deep --newuse world and then run your script?


Well, my guide actually requires the user to do so before invoking my script.

However, in your case this might actually be an overkill, because if the USE-flags changed by the hardended profile were all that changed, you can save some time by using the following approach:


  • Do the emerge -auvDN world while still being attached to the old profile.
  • Switch to the hardened profile. But don't do any emerge --update yet.
  • Re-emerge GCC as
    Code:
    emerge --ask --update --oneshot --newuse --deep gcc
    This will use the new USE-flag for the new GCC, if there are any. Look at the outpout of emerge whether any USE-flags did actually change. If there are none, and CFLAGS in /etc/make.conf also didn't change, there is no point in re-emerging GCC, and you can skip this step.
  • Run my script. It will recompile anything else, according to the hardened profile.


Update: I have also updated by guide (as of version 1.11) to support your scenario directly.


Last edited by Guenther Brunthaler on Sat Sep 09, 2006 7:14 pm; edited 1 time in total
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 217
Location: Vienna

PostPosted: Sat Sep 09, 2006 6:20 pm    Post subject: Reply with quote

juantxorena wrote:
why don't try to contact a developer to talk about this? Maybe the script can become an official part of Gentoo in the future.


While this may be an option for the future, I want to see much more testing before I will actually consider doing this.

While the script itself has not changed since this thread has been started, the accompanying guide serveral times has.

As more and more users were testing my guide, a couple of them have also suggested various improvements, which in turn led to several revisions of the guide to take those improvements into account.

I want this to get on for some time before I eventually might dare to contact any Gentoo developers.

juantxorena wrote:
Have you contact any developer? What he/they say about this?


I am not related to the Gentoo project in any way except for my commitment to the project. I'm just a normal user like most here in the forums.

juantxorena wrote:
Messing with the system like this is a dangerous thing,


Yes it is. That's also why I consider more testing to be essential.

Although it seems the guide actually worked fine with the people posting here, that says nothing about how well it will work in unusual cases of system configurations.

juantxorena wrote:
I don't want to break anything.


I can understand this.

But there is actually little reason to be afraid of: The script is actually a wrapper around a set of emerge statements, each one emerging another package. The script will not try to interfere with the internals of Portage and won't in fact do anything you were not also able to do manually.

The worst thing that could happen to your system is that the script stops before being finished due to a build error of some package.

If that happens, simply skip that package and let my script continue. Then, after all else has been rebuild, re-emerge the failing package again, hoping the emerge will then work better.

And if it turns out that my script can't continue, because that failing script is an essential dependency for other important packages, you can still revert to the usual

Code:
emerge -e system && emerge -e world
emerge -e system && emerge -e world
emerge -e system && emerge -e world
emerge -e system && emerge -e world
emerge -e system && emerge -e world
emerge -e system && emerge -e world


which some people consider to be a better solution. (I doubt it will, but I could be wrong. The only difference one will see for sure, is when looking at the next electricity bill.)

And if that standard method considered "safe" by most will also fail, you still have the last-resort option of doing a complete stage1 system reinstall, just as badpenguin suggests in his guide.

So - no matter whether my guide actually works for you or not, it will never damage your system to a degree that would not allow you to revert to any of the other guides around.

Such a case has not been reported so far, though.

juantxorena wrote:
the way you explain things, I think that you are acting like a televangelist or an infomercial vendor


Since when do televangelists or infomercial vendors explain anything? ;-)

They usually want you to believe something, which is definitively not my intention. If it were, I would only post my guide, but not any accompanying guides where I explain the ideas and concepts which eventually led to the creation of my guide.

So, if you felt like listening to an televangelists when reading my guide, I have to apologize: This has not been my intention in any way.

Also, as I am not a native English speaker, I will certainly not always be able to express my thoughts in the same natural way a native speaker would be able to do it in the same situation.

In other words: Please excuse my rather bad English!

juantxorena wrote:
This undermines you credibility.


To be honest: Being still a Linux newbie, I do not have any credibility yet. So there is hardly anything to be undermined. ;-)

So I'm doing the best I can to read man pages, HOWTOs, source files, etc and figure out how to do things in the most optimal way.

And when I find a solution to a problem that finally pleases me, but is different from the solutions I have found being suggested by others so far, I will share it with the community by posting an article.

Of course this will be done in the hope it may also be useful for others.

But the other reason is that is allows others to comment on and suggest improvements, which has shown to be a quite effective way for improving the quality of any guide.

juantxorena wrote:
It is in fact UNNECESSARY to recompile any package bla bla bla" or other similar quotes is not the best idea.


Well, perhaps you are right. It may not have been the most polite way to put that statement.

But it truly reflects my personal opinion about the issue, so it's at least not a lie.

Also, except for unusual scenarios where the usual "emerge -e system && emerge -e world" would fail as well, there have not been any serious problems reported with the most current version of my guide so far. (Which does not mean there were no problems reported with older versions of the guide. But I updated the guide in such cases in order to fix it.)

But up to date, my guide is still the fastest way to rebuild an entire system (assuming users have been updating their systems on a regular basis even before considering using my script; my guide will exploit this).

There are known situations in which it will not work, such as updating glibc from version 2.x to 3.x. But I won't expect such a major glibc update to happen within the current decade.

Also, in such a situation, all the other guides based on "emerge -e world" will also fail. You have to do a complete stage1 reinstall in such cases.

But otherwise, the guide seems just to work.

juantxorena wrote:
Personally I would wait to see what devs think about this.


I assume the Gentoo devs have more serious things on their do-to-lists than caring about such minor issues.

After all, my script is a mere optimization. It is nothing new or revolutionary on its own, it will only save you some bucks on your next electricity bill. That's all.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 4 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum