Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 6:05 pm    Post subject: Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS Reply with quote

Hi all,

(You are reading version 1.19 of this guide.)

ABSTRACT: How to recompile your entire system / toolchain with MINIMUM PROCESSING TIME EFFORT.

If you plan to do a major compiler update for your Gentoo box, such as from GCC 3.3 to GCC 3.4, or from GCC 3.3/3.4 to GCC 4, then read on. (This will typically be the case when upgrading your Gentoo installation to the new 2006.1 profile.)

The problem is that in those cases the C++ ABI of the GCC changes, which requires recompiling most of your system.

In other words: You have in fact to recompile your ENTIRE SYSTEM in order to be sure no "undesired" side-effects may arise from your compiler upgrade.

This is especially true when using KDE/QT and other programs/libraries which extensively make use of the C++ language.

And here is the reason for this posting: While I have found many postings how to best recompile your entire Gentoo system in such cases, or how to rebuild your "toolchain" best, I found none of them being optimal.

The problem: Actually all of them require much more time to execute then is really necessary.

Because in my opinion the amazing truth is: It is in fact UNNECESSARY to recompile any package, including the new GCC, more than exactly once when rebuilding your entire system! (For those who cannot believe this, see my argumentation in the related article http://forums.gentoo.org/viewtopic-p-3548628.html#3548628.)

Using my guide (and the accompanying utility script) presented here, you will be able to rebuild your entire Gentoo system with an absolute minimum of effort.

Compared to other guides which suggest recompiling the entire system up to 6 times "to be sure" (I will prove them wrong - their approach is not safer than my approach presented here, it only takes more time), my approach can save you days or even weeks of processing time!

And as weeks of unnecessary processing time equates days/weeks of unnecessary power consumption as well, this will certainly save you a few bucks on your next electicity bill.

I feel obliged to mention at this point that badpenguin has also written a guide how to recompile your entire system, taking about the same running time. Although not for the faint of heart (because it requires a stage 1 reinstallation of your system), I consider it to be a viable hardcore alternative to my guide. You can find it at http://badpenguins.com/gentoo-build-test/.

OK; let's be short: Here is my guide as of 2006-12-05:

===================

Recompile Entire System Helper

Copy the script from the end of this article and save it to a text file. (Go to the section "HOW TO OBTAIN THE SCRIPT" of this article to get it.) Be sure to follow the instructions below before you use it!

Script version as of $Date: 2006-09-26T04:22:55.640944Z $

Written in 2006 by Guenther Brunthaler

This script will generate another script to be run by you. That other script will then recompile each and every package of your whole current system in the correct order.

This will typically be required on a major GCC upgrade.

IMPORTANT: Do not execute this script before all of the following prerequisites are met:

  • Portage tree is up-to-date (emerge --sync)

  • Your system is up-to-date (emerge --ask --update --deep --newuse world)

  • gentoolkit is available. (The script uses it.) If you are unsure, just do an 'emerge --update gentoolkit' and it will be emerged unless it is already installed.

  • OPTIONAL. Review the /var/lib/portage/world list of installed packages and consider unmerging packages you are no longer using. It's a good time cleaning up your packages: The less packages are installed, the faster the recompilation of the entire system will be finished.

  • OPTIONAL. Now it might also be a good time to run "emerge --ask --depclean" in order to uninstall packages representing outdated old dependencies no longer used. Be sure to read the manual page for emerge before trying this!

  • Run "revdep-rebuild". If there are problems found and fixed, repeat running "revdep-rebuild" until no more problems are reported.

  • OPTIONAL. If you want to change your Gentoo system profile to a new one using "eselect profile", it is now also the right time to do it. Also do an env-update after changing the profile. (Be sure your kernel version is supported by the new profile! If necessary, upgrade the kernel, install it, reboot and continue following this guide only after that.)

  • Do an "emerge --oneshot --update --newuse gcc". It is quite common that emerge will detect that the current GCC is already up-to-date at this point, and will therefore emerge nothing. (But for the less common situations, this step will prevent troubles later.)

  • At this point, the new compiler you want should already be *installed*. (No packages need have to be recompiled with it yet. It also need not be the currently selected default compiler version yet.) As GCC allows multislot installations, it is not a problem in Gentoo to have both your current and a new compiler be installed at the same time. This guide even depends on that fact!

  • If all the above conditions are met, and no more packages need to be compiled in order to have an up-to-date system, set the desired new system compiler as the new system default compiler using "gcc-config". (This will make the new GCC 3.4 or GCC 4 the new system compiler.)

  • For those who have never used gcc-config before: "gcc-config --list-profiles" displays a list of all GCC versions which are currently installed on your box. The compiler you are currently using is marked with an asterisk. In order to change this "current" compiler to a different one from the list, remember the number between the brackets on the left side of the line which contains the compiler version you want to use. Then use "gcc-config <number>" to change the system default compiler to the list entry <number>, where <number> is the the number you remembered in the previous step.

  • Do an "env-update". Then run "gcc-config --get-current-profile". Verify that the new compiler version is really displayed as the current system compiler.

  • REBOOT! Many troubles when recompiling the system with a new compiler are due to the fact that the updated environment variables may not have been propagated to all active shell windows which you might be using for command input. A reboot ensures that every process in the system now uses the updated environment variables.

  • Are you using ebuilds which install kernel modules? Typical examples are special ALSA drivers, closed source video drivers, virtual machine networking drivers, add-on encryption engines. Not sure? Then better assume this is the case, and do this: Optionally update your kernel first. Then recompile your kernel with the (newly installed) GCC. Or, if you are using genkernel, just re-emerge genkernel. This will eliminate potential build failures later when the kernel module ebuilds will be emerged as part of the system rebuild. For instance, Karsten1973 reported build failures for the nvidia drivers when following an older version of this guide which did not include this point.

  • Download my utility script now if not already done. The link can be found at the beginning of this article. A good place where to put the script after downloading may be /usr/local/sbin/. (This is only a suggestion. Any other place will do as well.)

  • Run it!

  • The script will do nothing dangerous on its own. Not yet. Instead, it will use gentoolkit's services in order to construct a list of packages to be re-emerged when recompiling your entire system, and also determines the correct order in which those packages must be emerged. It also filters out unnecessary or duplicate emerges.

  • Then it will construct a shell script which, when run by you, will emerge all the packages in the right order until every package in your system has been recompiled as required.

  • The generated script will also allow to be aborted and resumed at any point. That is, you do not need to keep it running for days or weeks. The script remembers the last package successfully emerged, and will skip any packages already recompiled when run again later. This allows incrementally rebuilding your system. You can even shut down your system and continue recompiling the other day!

  • Run the generated script to actually start recompiling your entire system.

  • Run "revdep-rebuild". If there are problems found and fixed, repeat running "revdep-rebuild" until no more problems are reported. (Thanks go to devsk who identified possible problems when omitting this step.)
  • Reboot. (Just to be sure. It's not strictly necessary.)


OK, that's the guide so far.

If my guide served you well, you might perhaps also be interested in my article http://forums.gentoo.org/viewtopic-t-495997-highlight-.html where a promising new approach to prelinking is discussed.

Now some warnings and comments about the above guide. Although not necessary, it may be wise to read them also.


  • The script is now tolerant when packages fail to compile. In this case, the failing packages will be skipped and their names will be recorded in a file. Then, after all remaining packages have been recompiled, the script attempts to recompile those failing packages again. This will be repeated until all packages have been recompiled, or until none of the remaining packages could be recompiled successfully. This new feature allows an interruption-free rebuild of the system, without a need for the administrator to periodically monitor the build progress. The idea for this very useful feature was contributed by count_zero. Thanks a lot!

  • A different log file containing all of the screen output from the generated script will be created each time it is run. Together with the new interruption-free rebuild behaviour, this effectively turns system rebuilds into long-running batch-jobs.

  • The generated script assumes your portage tree will remain in a consistent state during the rebuild process. So please don't do an "emerge --sync" again before your system has been recompiled completely! (This is because the generated script contains specific package version numbers which were extracted at the time the script was generated. If the portage tree was updated before the script finished, package dependencies might change and the assumptions of the script regarding package dependendies might thus become void.)

  • In former versions of this guide, I strongly recommended running this script from the text mode interface, without an X-server running. Due to the experiences reported by Lloeki in his posting http://forums.gentoo.org/viewtopic.php?p=3553978#3553978, I consider this no longer to be a prerequisite, as you follow the general guidelines from the referenced posting.

  • Repeating from that post, those guidelines are: If you are heavily using KDE or other C++ based applications, start all such applications you may ever need during the update before running the script from my guide. Keep those applications running and use them until the update is finished. But don't ever close them, or there are chances you won't be able to start them again until the update is complete! Then, after the update, quit all applications, log out and reboot. (The trick here is that shared libraries which are already loaded will be kept im memory, even if their on-disk files are being replaced by different versions during the update. As long as you don't close those applications, the old versions will remain in memory, and all will remain consistent.)

  • The above trick allows you to keep working with your applications using C++ shared libraries during the update, which normally would run you into troubles. If you have a second machine to work on during the update, it may still be better to shut down the X server during the update and keep the machine running in text mode.

  • If you don't know how to get to the text mode, here is a short guide: First log out of your X session if necessary, so that you see the X login-widget of your kdm, xdm or gdm. Instead of entering your credentials here, press [Ctrl]+[Alt]+[F1] now. This should switch from the X login screen to the text console where you can log in as root. Then run "/etc/init.d/xdm stop" in order to shut down the X server.

  • Although the approach presented in the above guide greatly reduces the amount of time required to rebuild your entire system in comparison to all the other guides I am aware of, rebuilding your system will nevertheless take bloody ages. (At least unless you have a distcc compiler farm at your disposal. Or unless you are a "Bot-Master", controlling thousands of hijacked Windows PCs serving as distcc-slaves... ;-) )


    So far for my comments - for now.


HOW TO OBTAIN THE SCRIPT

Here are the instructions how to get the utility script mentioned in the text of the article above.

  • First select the text between (not including) the "---BEGIN SCRIPT---" and "---END SCRIPT---" lines (can be found at the very end of this article text) with the mouse.

  • Then copy the selected lines into a text editor, using the mouse or copy/paste.

  • Save the copied text to some file, such as "genscript.sh".

  • Launch a command shell and go to the directory where you saved the text file.

  • Run the command
    Code:
    sh genscript.sh
    This should decompress the actual script and leave it in the current directory as a file "recompile-entire-system-20060926".



Final remarks:

I wrote this guide not only as a small contribution to the Gentoo community, but also to the environment.

Yes, to the environment! Thousands and thousands of Gentoo users, all wasting weeks worth of processing power when following inefficient "recompile entire system"-style compiler upgrade guides, will waste an enormous amount of kilowatt hours of electrical power.

This energy not only costs money, but will also have to be produced, where it will contribute to even more pollution of air or water.

And that additional amount of completely unnecessary environmental pollution can easily be avoided by following the above guide... so that's why I wrote it.


Greetings from Vienna/Austria,

Guenther Brunthaler

==============================
Code:

---BEGIN SCRIPT---
#! /bin/bash
# This is file "genscript.sh".
die() { echo "ERROR: $1" >& 2; exit 1; }
echo "Decoding attachment - please standby."
which openssl > /dev/null 2>& 1 || {
emerge --oneshot --update openssl || {
die "I need openssl for base64 decoding!"; } }; {
while read LINE; do test "$LINE" = "exit" && break; done
openssl enc -d -a; } < "$0" > script.ar || {
die "Error in base64-encoded text!"; }
ar -x script.ar || die "Cannot extract archive!"
N="recompile-entire-system-20060926"; md5sum -c $N.bz2.md5 || {
die "Checksum error in extracted bzip2 file!"; }
bunzip2 -f $N.bz2 || die "Corrupted bzip2 archive!"
echo "Script extracted successfully!"
rm script.ar $N.bz2.md5
exit
ITxhcmNoPgovLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA4MCAgICAgICAgYApyZWNvbXBpbGUtZW50aXJlLXN5c3RlbS0yMDA2
MDkyNi5iejIvCnJlY29tcGlsZS1lbnRpcmUtc3lzdGVtLTIwMDYwOTI2LmJ6Mi5t
ZDUvCi8wICAgICAgICAgICAgICAxMTU5MjQ1NDAwICAwICAgICAwICAgICAxMDA3
NTUgIDQ0MjggICAgICBgCkJaaDkxQVkmU1kWFB8ZAAULX4BSeX///7//79//////
/2AS/3tHS7sWlYVI7DsqwG5umjOihrIhOWRUtmUdAykKVLbG5g3YcNNECmTQYjRG
0Jpo0mARqeNFTwUZoYptkiHqGIwGgQARDTVPTIJqfqYp5TR5RoA000AAA0AM1NBw
NNNNBoaGhkaAZAGhoDTRkAADCYgNBISECJqabU9QnhU/RT1HpMjTR6aTTRtQAGg/
VPSGjQbUBwNNNNBoaGhkaAZAGhoDTRkAADCYgNBIkE0BAmI0TAEylPwk1D9U2mSa
GmaTRoPIgehHqaWomCAaCqSbGwGNtsTGmMTBgwbGJpgDGm0MbE29fm/nPWQ9lHG/
ro5F/yfGKvAseigl8leYkAqHMC9lYAPlo/3B6cxuIfudv/T6LWorn3Mw/avQ/nkd
nhtnXUtJIs5q5CQjTcYNpxQbbTf2qqkQY5HHBjGyONsbDjxtZOzcG2zaUevnylrM
b1P9/PvXKDDXejGoTNScvPn+nbuPFO0/L/da3voQZz3PhBDJ7IPFJ0sklmU0Nm7h
Y2sXNdZfszv1cL4MZ0t8Ii7NVVmVpeBX0SvimzNTtZEerzJIEujVCQoCsZAi/OAE
z+3/G7cdmqGPv+IfVV4yok5d6SjfKf4wiU0oK33Y3ard+y8unaWYSa72rwG0Nq60
SZJlWpHP/jvrmYNqCCCTJbjO9ldM0TwB0gsSL1b/hee93Dr8pTVKIimwZ7aVMDCw
p5uJnu2CUSRQEAyR0Ne4GvcnHCFVYxuPZRME3GZhSpVeZk68FuYYbUKeQ+7ojnYV
lEtpn2eUKhh486CObEvTmXCmgPcNp3znCE4V1jSyEv9blHHWLJQDkIkDHLB2dOTf
1cuumHUjcux9uim76W9B1xEAvxa14dT0CRrpXr7u2p/Ltfr7fz686rFA1YafnIlm
r1tygx7eOSk6fQKrooIk2OzPlaYVJD8xuB9+952wsvuzRwlHqMgpRc5XgezGzht1
psZwWezWFaAKzp0KNIcN6MQLiu4VG4FYsWmdrVxTO+QVZ2Ae0nsMD9HQcBsflu+F
U3pUWLU14ILxOa+9pSLhlTwwT1KIiMo/u4MTCMzEXmsh2rc6OQ0ikAamULgwY6r0
+sj57wrjZS3+k4MGNEjBMrVwwc0kHrZP4atkRmZXQ0QkWHGBOiIoktOgBM+keY9Q
D4vMac5qDtMOg4G6K1uLdDeGmKkmwIUeSyqM8sPBcdy7FKVSSfic7b3XBdZ6JCTq
Uy7tacVOKjS6ScJfRK6/gP3xk1mFxsnrrrJgKI6/F10sDUFlnWWbcR8dQ1URhRrT
/dAkdjQyHqh6QCWRZDU65x1I55WgdmJ1dKja2/lp8LT9BDfkmGZUpvkUPwOavoxQ
d1rDFOk81CsIDEFx5M9hjnnuAcQrm3IkPUm25SocsApBwEYs+JuSq+Od1B7SMJZ1
uhI2EiYaGxENq98DZDSwcYY8hpjpGqyAPwuhm/g1Mpo10CNdTxZequbQkk5iOg2y
AElg5AHClcHaEZ78DoZthwX1Qg8eRlxxFGvSftA5vklDQratCgzbLbMtz54clQit
Ys6h5OvXHI3YrU5eUFZrNqSIDyAxPZ+GiYcAHIaUF9JLZT9iV2dqMAHPXrI2yA2z
6B3AidRW2836MFA4iL27p0VfUpltly2uyTzd2US6fKTgTXo6nYWrYZZ3L4Rex06o
tNRj53LPH9ueckAlEKEAEgAEoASDlWJ7tj2NjenFYcBcbxFp7bt+xRjNK30CDUxD
aFd+pEDBNgLs9LE0B0NVaC5x688l7fx2PGWIiEihlS5DDe1cG0wY2i40iAwYxjSY
vxCmK/ows/+HyxsytB5wX8cERiOj/Uj8hAaMoOzQVbuH/rjNNh3PFn0O2s5go0aK
4EabZQ4D2Grlob4Npigu7gRbBqPLXoSVfztav9tuNQc/CWLB/hrOzrKI2sET7tjX
/q+cA429m8HJsmo1dzuwkw/adF1pVxsg0ERKQP41L7MaeoDrHrelrkUffSLyNOL0
vDv+fz3e24PMRgI4/l624LVGEjcDDYIupKILCFI9gz0HG1r23Z7vl7t0eeAR8pvU
wxSeo+y59up/x4rmnsonWbD0zJsGkbLBuceLHhW7sNi5N4Mx78OSNnb4rfFhq/pp
sDbAMa9nY8UFIW+1U0yueqVl8La2FgChAK4ifrZRJ90FwubRvRckbgvZUcZkiIJ7
4wTm+PCOTnkIB0Pt04+Ad33Zk9nXtaqtQL3in0RZApg+AWA9F9FHRiiyD60Nw+XR
CyeZ4ZhvlNzCtFB7+yl4zU9/BTbPM9LWquBbo2QU39lCj6luScHaRyXq4J3z4ekQ
J9+CFNSbOkunQGn46bZ8Tdbq/Td2xwjqf1cMBPw64ynQwhxnXwMIoHqjeQ4BepH+
3Wsu3YMGTmgjcJDmQC3mCdKRCV7qf6SKe9CPGdQVKKHsPxeU+9hWqKyF7FgtFI8L
td2OdTuXxW8HjtVQXo4QPpgSASlNFKNQUvCdUk3HxIOMgf/5/Y5N8HRoXVewPkb6
YEjdpguf0yZevzcmg9Mpamgamh2rKZwWed7quSW2iEG24XagcsR6D31dAZ73qduP
NRLxJ1nvMJrRK62SgbN4ScUQGt/meVVfOWTqAI+WjwVonKAD7ejWnwJQpv1KE2kf
dzYgGsxZt3ftG257yyLwrIzJn3xLMG1b/AJJHvmvQ5pBdb7xOoeRSvUvj0MNcDi/
JsyvZ7wnvIG6QcgU7CuEHS7016r6Yf1PzCQ2AxgNfYQH6394LtvDmxPvD2LnUw/G
Xma9fzSUTOIvhEDf70H7KdK/bnYL2xpVvPpabObmWNO7vrVRhINokBkRAZE/Yolk
FxrWf2oIBWDEJroy1PJP5sFAw04oX/syTMliN7UiRk1vLw4OU9/QhfeCxoHyPPiC
4oNSdMLvnlqSzLmSNNgIkKCQZyVqInfjYL7gMyLgSaz2Guu0MAxoa6QYoxaLTdpg
gZ8jJLMLcHRszRig6LukqtVwjJY/3UxpD1PFaLswNmZYuGT60vwQck5UHrSu5jYI
wpXAiOKOTJk7gsguslRU5VU0SNWGQFD4/hI/wXPTPREA1rXHBZd0GciOlG0NTEcj
d2t15Q7azmCKD5bDfp+lVAcovk2slIIII2KPD3M7CPDeeu2XV67jm3e8a+pdpHff
AM2zWz2I/KNH/AMw+byXYnCcfFUPrPqAZFpdeYi/PDy/hOh5Aev/VHutATNXEvmh
EaB/BfKw+FoV9540dxDRGhvo6Z+ZXDci9ls6XrKN7bosnViLuQQjXpRptIopTXpx
RzSD7XpvvL1SihbBrfZTJGlMbDMPqUgnbQ+6tHNNYHO8Ubgtdor0VO338ZJI6XZb
amD4/uXz3f94ZJc6XNQRjxkNVx/YQGFOsNA3VcIb0zMLWpDaZkp6ft+9uARYSPFh
jWllGJtjf6IQgwI+g87Q2hFGzzTMsGTo3veOzMB7zUV2uuBKZSB0wbE2NMuzDOkY
rr/2S9LWve0mhvLsZsrYRAV5YhQq99lRAbGZsW5O5Fx923IHw4y7FLDIvQpDXeW8
adffxTZiQ7YzMIHXq5BTGrDfqxcRStS3OR4WEhJ5OiEqFiowghB1bIIScJzR1pUd
ASgEB2YB4WiAkJHLZmo+U2QLE2pMOlIB9H4tOVVswTNUs2zAYPHRr0usa4RWxr6Q
2BPE3piI0X5Gigoer/keJHjflegwRGLBg9TBThpWlugLbccu87VDMydik/DIZkxU
ebIdHJZtclEJMTVCRr3LDA27NzioZIzaDFS8pElu98l01uA3ByG6KJShw2SiCwfH
ejcg5b06NjgXUsrrpqpgMiHkEleZPFVfBnFliZxgKEpcRWqerufvXOi6rs0MNabJ
rXgm3vga0onzTpdbPRXDJMq2ElXdMwUi+4mLHY1rhcDI6eRM8IiZo2c3OeGcYl7c
qkEcZ8cUabBluh3RGiQgD6YRjZqH3osty+otD9WXMNLGQp222KGixEf2yn95xJmH
rCmep1HfiaaREEIBs5VLz+OuWtlAYVMVIaUIE13z1VGD+NKX8/eFbigD31QgGsYo
BXTy7kxqEUYNsciHwiPgssg5kee9GrcEUxsG0NpD9K6j5DhuMIMPy9ziBqpuoMMV
Cm0yFpRcWJBLAzauwa0tz6p9wrkIjyLoY4D2hv0DHNHmIMKNemo7vDVESfgOP4y8
qtmointYuRsc+O+35DiccIwUEgQJ+xwuNsWJxEEh83w6PBaIqMDzv9F1zfbDThbE
aCYEszp6oDxb0rGwl7PxTV1taMfczC/LtTuAaMzXWAwaBjZ79o7kJTOHQYIyQdGQ
0mw1raazRc4xlVTifoOzRMmQcpFNwEDpvigByZRkI0Z8A3LVseAg5HPDY5zR4ME2
XRANh1I2BKJjKPS/wgtVZILrySR1tGpmtot+QvkkSF4Ns0Utq1GG2AyOFruBCKsJ
W98lYwChsaRAx/jZCewGiZ52sHzPnDatdjCBRemS76KjNGV4TRmx2gMGfJw0ygNU
HVQsHcm2SCp9SnFLjyu3rqrgVhFUqiQ4C94Wuve35vdLu93UTkiJxMfZSZijU1sj
0yDYNXCWZSZcOFi9mXYXd3RaDTVOgfohZ2VsWVqJ8QMuyqsvcsaBbSuzLY+zFfVh
4GiLYFYHcmEMWVwoAlM2PVImkizKoGHJi07WGKwGd2vpGlkDR3XIGMhpvRjYmBBi
30aqlaMtVk0M8mP3FGYdq6kdbFwYhQSkz01f1yRp9kFR5udvBacO9eJlMSnQP1wH
8lff9PwVDImTYYpI5liMG083CzYag2britCBYGrXnqu8Uwr6KEd4G0W4l8mI3bu/
3EzFzLNo/mpwDbyD4quvcG3SyAmdK5u3VRv4eaVtPYBnP/y+petwZ67mvV3l5D6p
kRTsqtoon1JVeFiuPGwWxyXMI9LF2ubjExquoxSzfhY0GjKe8ySC1DQDGZiJmV5U
9j9D64kPX9IRgWSvYDYGiYtROSJpge7tx5M3GvMaBGme/A0FzxQ4AyDA3uRobId6
TD/7aXeVaEhhiu0fgIhhQml3wLmlhjochwVrUihejpPMdtjsYdkiNtVtjvlxqwbo
MJEqOPWZUUwizXeKR4HF6RxL2u/kG5sMQxAzQlu+tOtLuZhJZ3sIvF5GUUrVVS2I
DIFQucVuGOEkt+KUiAkkcrmsZTdMrVFZG2AmSEDDXBu04VVaOtpMmK+HLDbHpzWo
r0Ztee31kT3yPCq+Xj2Uw3atEePbAl3CXE215whQOQMpLioWBOZvhmVaV3UJHKZ9
eEBZMe3tlHl9uhUSxGsgctJkyInpf7+SUKnBKFFlvUhZkQeW3E/L00P2soRlkubB
h4at7cBzQzYFc0IVAVgWsIokTCdwaXeyoSmQqsXTfCTxzMSCdgjhaDymYfC7B0EI
pmplMwKTCeOxoKqQUlFzUptv1sWFcTuk4h4PBVFGVbMBf7jA0MFCNiMp6pK5qSvz
uiawGel2GAwY2O/eeYpLmLYrJWWC1LevkzR0GSGy5ZAkNTzDVNBrcxyyKnjWmq6d
XpaxkUE2mQcNDuYYqb5NQxaNHXKDo1kj26UOFFtSxEnxZ9FeuW83DDbcjcrhgV0G
3sS3wC/Q1wpN/mXz+SVvPqCUy+iBHOOBwLJECdB/YGPHxcrqdKKi0XFQMJWmm1Ay
djskdRnhOk78zyIeB1+Pj1GFMuk1urtpooJUWvVDxd1Q39hEWFD1te9hHmEBVDDx
dGBRlRqWBC2mA7H3TiYQXVzPH9Oua2MIYuHRzCDk5BAdI5nEtNTp9Dp6zkOtbBKV
H9l3LYBiQL2tH8LbQ8+osulggvA5z9ww/5+TE6BlEiSX+JZB7V6h2dUT+eJB4gC3
4BZc0seSQiogcPkt8SBh7Zf+LuSKcKEgLCg+Mi8zOCAgICAgICAgICAgICAxMTU5
MjQ1NDcwICAwICAgICAwICAgICAxMDA2NDQgIDcxICAgICAgICBgCjJiZmQ1Mjdl
NTQyY2MzY2Y5MDY0MTBiNDhiZGRmNDBkICpyZWNvbXBpbGUtZW50aXJlLXN5c3Rl
bS0yMDA2MDkyNi5iejIKCg==
---END SCRIPT---


Last edited by Guenther Brunthaler on Tue Dec 05, 2006 6:11 pm; edited 19 times in total
Back to top
View user's profile Send private message
Nick C
Guru
Guru


Joined: 18 Mar 2005
Posts: 526
Location: Portsmouth, England

PostPosted: Fri Sep 01, 2006 6:42 pm    Post subject: Reply with quote

can you not just post the script in plain text inside code bb tags? Would save anyone wanting to follow your guide a bit of hassle. :)
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
www.monkeydust.net
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 7:06 pm    Post subject: Reply with quote

Nick C wrote:
can you not just post the script in plain text inside code bb tags? Would save anyone wanting to follow your guide a bit of hassle. :)


Sorry for the hassle!

Initially I wanted to do just as you suggested.

But then I had concerns that the whitespace and indentation of my script could be ruined by the BBCode formatting and/or careless copying/pasting.

There were also data integrity concerns: What if only part of the script was copied? This script does work vital to the future health of the system. It must be assured that its contents are intact before starting! (So I added an MD5 checksum file as part of the base-64 encoded text.)

Another problem is the script length: It is relatively large in uncompressed form, so I wanted to bzip2 it in order to get much smaller.

The optimum solution would certainly be to host it on some web server, and only provide a download link in the article.

But unfortulately I'm already quite a bit over quota with my ISP, so I wouldn't like to put it on my own web page.

However, if someone reading this has some bandwith left to waste, he/she is invited to host the decoded script at their web site and send me the download URL.

I would then thankfully included it within my article.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2726
Location: Bay Area, CA

PostPosted: Fri Sep 01, 2006 7:11 pm    Post subject: Reply with quote

please post a text version, no matter how long it is. A useful thing is never too long to cut&paste....:)

I was hoping someone actually wrote such a script. Thanks for doing that.
Back to top
View user's profile Send private message
Nick C
Guru
Guru


Joined: 18 Mar 2005
Posts: 526
Location: Portsmouth, England

PostPosted: Fri Sep 01, 2006 7:14 pm    Post subject: Reply with quote

aslong as you use the code tags whitespace will be preserved
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
www.monkeydust.net
Back to top
View user's profile Send private message
Earthwings
Administrator
Administrator


Joined: 14 Apr 2003
Posts: 7750
Location: Karlsruhe, Germany

PostPosted: Fri Sep 01, 2006 7:18 pm    Post subject: Reply with quote

Code:
$ ./move.pl
Encoding, please wait...
Generating move script... done.
$ ./move.bash
Moved from Installing Gentoo to Documentation, Tips & Tricks.

_________________
KDE 4.12 - Get It While It's Hot!
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 7:39 pm    Post subject: Reply with quote

devsk wrote:
please post a text version, no matter how long it is. A useful thing is never too long to cut&paste....:)


Actually, the attached base64-encoded "attachment" is already a script! :-)

Look more closely at it: Although it undeniably contains a bunch of base64-lines at its end, there is bash code at the beginning which decodes the base-64 stuff at the end of the script, decompresses it and verifies the md5 checksum.

So, actually you have a script which you can copy/paste, and run.

You don't have to decode the base-64 stuff by hand - the script will do it for you.

If I posted the uncompressed script instead, you would have to do exactly the same: Copying, pasting into some text editor, saving as a file, running the script.

Only that the text to copy would be longer than it is now (because it is compressed).

Plus there is the the security bonus: The current version verifies its own data integrity. This would be hard to achieve with a fully expanded text version.

devsk wrote:
I was hoping someone actually wrote such a script. Thanks for doing that.


Actually I wanted the script to get even better; namely an interactive, dialogue-based version which asks the user questions and performs most of the "preparation"-steps of my guide semi-automatically.

But then I could not get it finshed in time, and decided to choke on the interactivity: I definitively wanted to post the article before the majority of Gentoo users have switched to 2006.1, because otherwise they would have to use the old, inefficient guides that are available at that time.
Back to top
View user's profile Send private message
MaBu-Gentoo
n00b
n00b


Joined: 28 Sep 2005
Posts: 22
Location: Slovenia

PostPosted: Fri Sep 01, 2006 7:49 pm    Post subject: Reply with quote

I have put the script on pastebin. I think it is a great script, but I have one question, when should I update glibc?
I konw have sys-libs/glibc-2.3.6-r4 and the new one is 2.4-r3.
_________________
[IMG]http://img101.imageshack.us/img101/2447/gentoo27li.gif[/IMG]
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2726
Location: Bay Area, CA

PostPosted: Fri Sep 01, 2006 7:55 pm    Post subject: Reply with quote

Guenther Brunthaler wrote:

So, actually you have a script which you can copy/paste, and run.

You don't have to decode the base-64 stuff by hand - the script will do it for you.
I understand all that...:) this is not the first time I am seeing something like this. But with the whole script in the page, I as a dev, can easily look at it and make a case for whether I want it or not. I can review it without copying it and then decide that Mr Guenther has done a good job with writing it, has handled errors and corner cases well. Even make suggestions to the parts which I think could be done better or in a different way. Its just that I am too lazy. And trust me there will be others with same problem.

code tags do preserve sanity and this is the standard way of copying scripts around here on these forums.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2726
Location: Bay Area, CA

PostPosted: Fri Sep 01, 2006 7:58 pm    Post subject: Reply with quote

MaBu-Gentoo wrote:
I have put the script on pastebin. I think it is a great script, but I have one question, when should I update glibc?
I konw have sys-libs/glibc-2.3.6-r4 and the new one is 2.4-r3.
thanks for doing that.... :D
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 8:00 pm    Post subject: Reply with quote

Guenther Brunthaler wrote:
Actually, the attached base64-encoded "attachment" is already a script! :-)


For the sake of clarity, I have edited my original article and removed all references to the fact that the script is internally base-64 encoded.

Those references were actually completely useless, as the fact that parts of the script are base-64 encoded is completely transparent to the user: The script itself handles the decoding; the user never comes in touch with the base-64 part of the script.

Also, it seems obvious to me now, that the mere mentioning of the term "base-64" has the potential to scare away half of the potential users right from the start ;-) - which really wasn't what I intended to happen.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 8:16 pm    Post subject: Reply with quote

devsk wrote:
MaBu-Gentoo wrote:
I have put the script on pastebin. I think it is a great script,


Thank you very much for this! I will edit my article and add your link there as soon as possible.

devsk wrote:
MaBu-Gentoo wrote:
but I have one question, when should I update glibc?
I konw have sys-libs/glibc-2.3.6-r4 and the new one is 2.4-r3.
thanks for doing that.... :D


This is easy: You don't have to. Recompiling the glibc will be one of the first things my generated script will do, and it will use the "current" version of the glibc as of the time when you ran your last "emerge --sync" before my script (which is one of the steps in my upgrade guide, btw).

Even if you decide to update the profile to 2006.1 (and not only update the GCC) this will be true: In this case, the newly activated profile will tell my script to use the new glibc version rather than the one from the old 2006.0 profile. (My upgrade guide also shows at what step one should upgrade to the new profile).

In other words: Dont bother; my script will take care of all the details.

Just be sure to follow all steps outlined in my guide exactly as described!
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 8:29 pm    Post subject: Reply with quote

MaBu-Gentoo wrote:
I have put the script on pastebin.


Thanks for your help, but unfortunately, the URL does not work for me!

But on the other hand, I have changed the "ATTACHMENT" from my original article into a script in the new version of the article.

So, the old version is outdated anyway.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 8:44 pm    Post subject: Reply with quote

devsk wrote:
But with the whole script in the page, I as a dev, can easily look at it and make a case for whether I want it or not. I can review it without copying it


I agree; this is a comprehensible concern indeed.

On the other hand, I am not willing to sacrifice the MD5 integrity check on my script for security considerations. (There can too much go wrong with copy/paste, escpecially for inexperienced users.)

But in this case the solution is simple: I will post an expanded version of my script for review purposes only, but the download version will remain to be the MD5-protected version.

devsk wrote:
and then decide that Mr Guenther has done a good job


I bet you would have been happier if you never saw that messy code... well, it's Perl code; and as such a good candidate for some code obfuscation contest... especially if written by me ;-)

On the plus side is that the messy Perl script only generates the script to do the recompilation.

And the generated script is a really plain and simple bash script without any blows and whistles.
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Fri Sep 01, 2006 9:06 pm    Post subject: OK, so here is the generator script for review! Reply with quote

Due to several requests, here is a line-numbered copy of the generator script which will be generated if the script from my article at the top is copied/pasted, saved and then executed.

This version is for review purposes only, please use the version from the article at the top if you actually want to download it.

The reason: Much can easily go wrong with copy/paste; and this is especially dangerous in a language like Perl where a single missing period can cause a script to reformat your hardisk instead of doing some useful job :-)

Which basically means, just carelessly copying/pasting can be dangerous. One must also be sure that the pasted text is identical (and not just similar) to the original text.

Which can best be achieved with checksums.

For this reason, the original script from my article contains a copy of the script below (in compressed form), and also an MD5 checksum. When the script is run, it expands the script below to your current directory, verifies its MD5 checksum and will warn you if the checksum does not match.

So please use the code below for review purposes only, but for the sake of security, use the version from the article at the top for downlading.

Code:

     1   #! /usr/bin/perl -w
     2   #
     3   # Generate a script which when run recompiles each and every package
     4   # in the Gentoo system.
     5   # This will typically be required on a major GCC upgrade.
     6   #
     7   # $HeadURL: /caches/xsvn/trunk/usr/local/sbin/recompile-entire-system $
     8   # $Author: root $
     9   # $Date: 2006-09-26T04:22:55.640944Z $
    10   # $Revision: 390 $
    11   #
    12   # Written in 2006 by Guenther Brunthaler
    13   
    14   
    15   use strict;
    16   use File::Temp ':POSIX';
    17   
    18   
    19   # Change this to any name you like.
    20   my $script= "recompile-remaining-packages";
    21   
    22   
    23   my $script_header= << '.';
    24   #!/bin/bash
    25   #
    26   # Run this script repeatedly (if interrupted)
    27   # until no more packages will be compiled.
    28   #
    29   # $Date: 2006-09-26T04:22:55.640944Z $
    30   # $Revision: 390 $
    31   # Written in 2006 by Guenther Brunthaler
    32   
    33   
    34   SCRIPT_VERSION="1.13"
    35   
    36   
    37   die() {
    38      echo "ERROR: $*" >& 2; exit 1
    39   }
    40   
    41   RM() {
    42      rm "$1" || die "Could not remove file '$1': $!!"
    43   }
    44   
    45   MV() {
    46      mv "$1" "$2" || die "Could not rename file '$1' into '$2': $!!"
    47   }
    48   
    49   CHMOD() {
    50      chmod $* || die "Could not set permissions 'chmod $*': $!!"
    51   }
    52   
    53   save_progress() {
    54      {
    55         echo "$VERSION"; echo "$OURGCC"
    56         echo "$PROGRESS $OK $FAILED"
    57      } > "$STATE_FILE"
    58   }
    59   
    60   item() {
    61      test "$PROGRESS" -ge "$1" && return
    62      echo "Emerging package # $1 ('$2')..."
    63      local RC; emerge --oneshot --nodeps "$2"; RC=$?
    64      if [ $RC = 0 ]; then
    65         echo "Package # $1 rebuild complete."
    66         (( ++OK ))
    67      elif [ $RC -gt 100 ]; then
    68         echo "Emerge failed return code $?. Aborting on user request."
    69         exit 1
    70      else
    71         echo "Emerge failed return code $? (will be retried later)."
    72         (( ++FAILED ))
    73         echo "$2" >> "$FAILURES_FILE"
    74      fi
    75      echo; PROGRESS="$1"; save_progress
    76   }
    77   
    78   
    79   BASENAME="${0##*/}"
    80   STATE_FILE="$HOME/.$BASENAME.state"
    81   FAILURES_FILE="$HOME/.$BASENAME.failures"
    82   LOG_FILE="$HOME/${BASENAME}_$(date '+%Y-%m-%dT%T').log"
    83   REPEAT="--4ydzd3yuhmsynbjr644bbfzx5"
    84   if [ "$1" != "$REPEAT" ]; then
    85      exec > >(
    86         echo "Note: Logging to file '$LOG_FILE'."
    87         tee "$LOG_FILE"
    88         echo
    89         echo "Note: A log has been written to '$LOG_FILE'."
    90      )
    91   fi
    92   OURGCC="`gcc-config --get-current-profile`" || die "gcc-config failed!"
    93   VERSION=; LASTGCC=; PROGRESS=; OK=; FAILED=
    94   {
    95      read VERSION; read LASTGCC; read PROGRESS OK FAILED
    96   }  2> /dev/null < "$STATE_FILE"
    97   if [ "$VERSION" != "$SCRIPT_VERSION" -o "$OURGCC" != "$LASTGCC" ]; then
    98      VERSION="$SCRIPT_VERSION"; true > "$FAILURES_FILE"
    99      PROGRESS=0; OK=0; FAILED=0; save_progress
   100   fi
   101   
   102   # Now the list of packages to be recompiled follows.
   103   .
   104   
   105   my $script_tail= << '.';
   106   # End of package list.
   107   
   108   echo
   109   if [ $FAILED = 0 ]; then
   110      echo "Success! All packages have been re-compiled."
   111      echo "Your system is now up-to-date with respect to $OURGCC!"
   112   elif [ $OK = 0 ]; then
   113      echo "Giving up: Could not compile any more packages in the current"
   114      echo "recompilation phase."
   115      echo "Look into the 'item'-lines of script '$0'"
   116      echo "to see which packages could not be recompiled."
   117   else
   118      echo "Partial success: The current recompilation phase has"
   119      echo "finished."
   120      echo
   121      echo "However, $FAILED packages were not compiled successfully"
   122      echo "in this phase."
   123      echo
   124      echo "Therefore, another phase will be started now, where an attempt"
   125      echo "will be made to recompile the remaining packages."
   126      echo
   127      echo "Note that this will *not* lead into an infinite loop:"
   128      echo "In the last phase $OK packages have been compiled"
   129      echo "successfully, and thus the total number of remaining"
   130      echo "packages has been diminished already."
   131      echo
   132      ME="$0"
   133      if [ ! -e "$ME" ]; then
   134         ME="$(which "$ME")"
   135         test -e "$ME" || die "Cannot locate '$0'"
   136      fi
   137      NEW="${ME}_$$.tmp"
   138      (
   139         IFS=$'\n'
   140         while read -r LINE; do
   141            test "$LINE" != "${LINE#item }" && break
   142            echo "$LINE"
   143         done
   144         {
   145            while read -r LINE; do
   146               echo "item $(( ++PROGRESS )) $LINE"
   147            done
   148         } < "$FAILURES_FILE"
   149         while read -r LINE; do
   150            if [ "$LINE" = "${LINE#item }" ]; then
   151               echo "$LINE"
   152               break
   153            fi
   154         done
   155         while read -r LINE; do echo "$LINE"; done
   156      ) < "$ME" > "$NEW"
   157      CHMOD --reference="$ME" "$NEW"; RM "$ME"; MV "$NEW" "$ME"
   158      VERSION="NONE"; save_progress
   159      exec "$ME" "$REPEAT"
   160   fi
   161   RM "$FAILURES_FILE"; RM "$STATE_FILE"
   162   .
   163   
   164   
   165   # Remove the largest common whitespace prefix from all lines
   166   # of the first argument.
   167   # (Empty lines or lines containing only whitespace are skipped
   168   # by this operation and will be replaced by
   169   # completely empty lines.)
   170   # The first argument must either be a reference to a multiline
   171   # string containing newline characters or a reference to an
   172   # array of single line strings (without newline characters).
   173   # Then optionally indent all resulting lines with the prefix
   174   # specified as the argument to the -first option.
   175   # For all indented lines do the same, but use the argument
   176   # to option -indent as the value of the -first option then.
   177   # If option -wrap <number> is specified, contiguous non-empty
   178   # lines of the same indentation depth are considered paragraphs,
   179   # and will be word-wrapped on output, resulting in a maximum
   180   # total line length of <number> characters.
   181   # The word-wrappin will occur on whitespaces, which can be
   182   # protected by a backslash.
   183   sub normalize_indentation {
   184      my($tref, %opt)= @_;
   185      my(@t, $t, $p, $pl);
   186      $opt{-first}||= '';
   187      $opt{-indent}||= ' ';
   188      $t= ref($tref) eq 'ARRAY' ? $tref : [split /\n/, $$tref];
   189      foreach (@$t) {
   190         s/^\s+$//;
   191         next if $_ eq '';
   192         if (defined $pl) {
   193            for (;;) {
   194               substr($p, $pl= length)= '' if length() < $pl;
   195               last if substr($_, 0, $pl) eq $p;
   196               substr($p, --$pl)= '';
   197            }
   198         } else {
   199            ($p)= /^(\s*)/;
   200            $pl= length $p;
   201         }
   202      }
   203      substr($_, 0, $pl)= '' foreach grep $_ ne '', @$t;
   204      if (exists $opt{-wrap}) {
   205         my $width= $opt{-wrap} - length $opt{-first};
   206         my $i;
   207         my $wrap= sub {
   208            my($tref, $aref, $iref, $w)= @_;
   209            my $buf;
   210            my $insert= sub {
   211               my($tref, $aref, $iref)= @_;
   212               splice @$aref, $$iref++, 0, $$tref if defined $$tref;
   213               undef $$tref;
   214            };
   215            return unless $$tref;
   216            foreach (split /(?:(?<!\\)\s)+/, $$tref) {
   217               s/\\\s/ /gs;
   218               if (length($buf || '') + length > $w) {
   219                  &$insert(\$buf, $aref, $iref);
   220               }
   221               if (defined $buf) {$buf.= " $_"} else {$buf= $_}
   222            }
   223            &$insert(\$buf, $aref, $iref);
   224            undef $$tref;
   225         };
   226         $width= 1 if $width < 1;
   227         undef $p;
   228         for ($i= 0; $i < @$t; ) {
   229            if ($t->[$i] =~ /^(?:\s|$)/) {
   230               &$wrap(\$p, $t, \$i, $width);
   231               ++$i;
   232            } else {
   233               if (defined $p) {$p.= ' '} else {$p= ''}
   234               $p.= $t->[$i];
   235               splice @$t, $i, 1;
   236            }
   237         }
   238         &$wrap(\$p, $t, \$i, $width);
   239      }
   240      for (my $i= 0; $i < @$t; ) {
   241         if ($t->[$i] =~ /^\s/) {
   242            push @t, splice @$t, $i, 1;
   243            next;
   244         }
   245         if (@t) {
   246            &normalize_indentation(\@t, %opt, -first => $opt{-indent});
   247            splice @$t, $i, 0, @t;
   248            $i+= @t;
   249            @t= ();
   250         }
   251         ++$i;
   252      }
   253      if (@t) {
   254         &normalize_indentation(\@t, %opt, -first => $opt{-indent});
   255         push @$t, @t;
   256      }
   257      substr($_, 0, 0)= $opt{-first} foreach grep $_ ne '', @$t;
   258      $$tref= join '', map "$_\n", @$t if ref($tref) ne 'ARRAY';
   259   }
   260   
   261   
   262   sub wrap0(@) {
   263      my $text= join ' ', @_;
   264      normalize_indentation \$text, -indent => '    ', -wrap => 79;
   265      return \$text;
   266   }
   267   
   268   
   269   sub pwrap(@) {
   270      print ${wrap0 @_};
   271   }
   272   
   273   
   274   $ENV{LC_ALL}= "C";
   275   my $home= $ENV{HOME};
   276   unless ($home && -d $home) {
   277      die 'Please set $HOME to your home directory';
   278   }
   279   $home =~ s!/*$!/!;
   280   substr($script, 0, 0)= $home;
   281   if (-e $script) {
   282      die "Please remove the existing '$script'.\nIt is in the way";
   283   }
   284   pwrap "$0 -", << '.';
   285   Recompile Entire System Helper
   286   
   287   Script version as of $Date: 2006-09-26T04:22:55.640944Z $
   288   
   289   Written in 2006 by Guenther Brunthaler
   290   
   291   This script will generate another script to be run by you. That other script
   292   will then recompile each and every package in the whole system in the correct
   293   order.
   294   
   295   This will typically be required on a major GCC upgrade.
   296   
   297   When the generated script will be run, it will log all of its screen output
   298   to a log file.
   299   
   300   Furthermore, the generated script will automatically skip failing packages and
   301   automatically attempt to recompile them again after all the other packages
   302   have been recompiled.
   303   
   304   IMPORTANT: Do not execute this script without also following the steps of the
   305   accompanying usage guide.
   306   
   307   The guide can be found at
   308   http://forums.gentoo.org/viewtopic-t-494331-highlight-.html
   309   
   310   Press [Ctrl]+[C] now in order to abort processing if you are not following the
   311   guide step by step. Come back and re-run this script after you have managed to
   312   get the guide.
   313   
   314   Press [Enter] now to continue if you dare.
   315   
   316   .
   317   <STDIN>;
   318   my $tmp= tmpnam or die "No temporary file names left";
   319   print "Collecting list of packages and evaluating installation order...\n";
   320   my @head= qw(
   321      sys-kernel/linux-headers
   322      sys-devel/gcc
   323      sys-libs/glibc
   324      sys-devel/binutils
   325   );
   326   my $r= join '|', map quotemeta, @head;
   327   $r= qr/ ^ (?: $r ) - \d /x;
   328   open OUT, (
   329      '| sort -k1,1 | sort -suk3,3 | sort -nk2,2 | sort -sk1,1 '
   330      . '| cut -d" " -f3 >> "' . $tmp . '"'
   331   ) or die "Cannot open output pipe: $!";
   332   my $n= 0;
   333   foreach my $f (qw/system world/) {
   334      open IN, "emerge -pe $f |" or die "Cannot open input pipe for '$f': $!";
   335      while (defined($_= <IN>)) {
   336         if (/]\s+(.*?)\s/) {
   337            (my $t, $_)= ($f, $1);
   338            if (/$r/o) {
   339               for (my $i= @head; $i--; ) {
   340                  my $L= length $head[$i];
   341                  if (length >= $L && substr($_, 0, $L) eq $head[$i]) {
   342                     print OUT "begin $i";
   343                     goto field3;
   344                  }
   345               }
   346            }
   347            print OUT "$t ", ++$n;
   348            field3:
   349            print OUT " =$_\n";
   350         }
   351      }
   352      close IN or die $!;
   353   }
   354   close OUT or die $!;
   355   open IN, '<', $tmp or die "Cannot open file '$tmp': $!";
   356   open OUT, '>', "$script" || die "Could not create '$script': $!";
   357   print OUT $script_header;
   358   $n= 1;
   359   while (defined($_= <IN>)) {
   360      next if m!^=sys-devel/gcc!; # It's already up to date!
   361      print OUT "item $n $_"; ++$n;
   362   }
   363   print OUT $script_tail;
   364   close OUT or die "Could not finish writing '$script': $!";
   365   close IN or die $!;
   366   unlink($tmp) == 1 or warn "Could not remove temorary file '$tmp': $!";
   367   unless (chmod(0755, $script) == 1) {
   368      die "Could not set permissions for '$script': $!";
   369   }
   370   pwrap << ".";
   371   Done.
   372   
   373   Script "$script" has been generated.
   374   
   375   Run this script in order to recompile each and every package in the
   376   system!
   377   
   378   By the way, the generated script will do this in a recoverable way:
   379   It can be aborted at any time by you, and will continue where it left off
   380   when you re-run it. (The package where the script was interrupted will
   381   have to be compiled again from its beginning, though.)
   382   .


Last edited by Guenther Brunthaler on Tue Sep 26, 2006 4:26 am; edited 2 times in total
Back to top
View user's profile Send private message
Abit667
n00b
n00b


Joined: 05 Nov 2003
Posts: 31
Location: PA

PostPosted: Sat Sep 02, 2006 1:26 am    Post subject: Reply with quote

Wish I had seen this a little earlier..now that I'm halfway into that stupid emerge -eav system and world crap from the guide. That always seemed so insanely inefficient to me...
Back to top
View user's profile Send private message
staffan
n00b
n00b


Joined: 31 Oct 2004
Posts: 29
Location: Sweden

PostPosted: Sat Sep 02, 2006 9:10 am    Post subject: Reply with quote

Abit667 wrote:
Wish I had seen this a little earlier..now that I'm halfway into that stupid emerge -eav system and world crap from the guide. That always seemed so insanely inefficient to me...


I know the feeling...

Personally I was just about to start that crap again but thought I'd browse around in a few random forums for a bit first. And I just happened to see this thread and grabbed the script :D

The recompile is running right now (I'm posting this from my window$ machine) and it looks good so far. Of course I have a rather old CPU (an Athlon-XP-something (I always forget)) and a lot of packages so it will still take a while.

I especially like the abort and resume features. We've had a lot of thunderstorms around here recently so sometimes you have to shut everything down very suddenly.
_________________
/S
"I'm reminded of the day my daughter came in, looked over my shoulder at some Perl 4 code, and said, 'What is that, swearing?'" (Larry Wall)
Back to top
View user's profile Send private message
stahlsau
Guru
Guru


Joined: 09 Jan 2004
Posts: 565
Location: WildWestwoods

PostPosted: Sat Sep 02, 2006 12:29 pm    Post subject: Reply with quote

hi,
thanks for the script, i think it's really valuable in this world of "i-recompile-my-system-a-thousand-times-to-be-uptodate" people. I can't say i'd need it since i've never encountered severe problems after updating gcc (i'm running 4.1.1 atm), but i think many people will find this useful.
_________________
Join the adopt an unanswered post initiative today
Back to top
View user's profile Send private message
KD-120RD
Tux's lil' helper
Tux's lil' helper


Joined: 07 Mar 2004
Posts: 149
Location: Hamburg

PostPosted: Sat Sep 02, 2006 1:45 pm    Post subject: Reply with quote

Hi

I have a question here: What is the purpose of switching the profile?

I thought gentoo is a meta-distribution. Just update your packages and configs (ie. baselayout) and there you are.
Besides I'm using gentoo for about two years now and as I recall I have not manually swicthed my profile.
When I take a look at my /etc/make.profile it is linked against ../usr/portage/profiles/default-linux/x86/2006.0/
Is that possible?
Back to top
View user's profile Send private message
grimm26
Guru
Guru


Joined: 23 May 2004
Posts: 302
Location: Chicagoland, IL

PostPosted: Sat Sep 02, 2006 4:17 pm    Post subject: Reply with quote

i don't understand the significance of my system profile matching my kernel. It's just default use flags and stuff, right?
_________________
"Blessed is he who finds happiness in his own foolishness, for he will always be happy".
Back to top
View user's profile Send private message
jure1873
Apprentice
Apprentice


Joined: 09 Feb 2006
Posts: 183

PostPosted: Sat Sep 02, 2006 7:27 pm    Post subject: Reply with quote

This script seems very interresting to me, but how much does this approximately take? days?
I mean, I have and sempron 3000 with 1gb of ram but if I remember installing gentoo from stage3 wasn't all that fast (I mean to get to kde)
Upgrading gcc probably recompiles everything so this is basically a stage1 install? Could I grab a newer gentoo cd and get binary packages for some of the files from that cd and recompile the other files to make it faster?
Back to top
View user's profile Send private message
CPUguy387
n00b
n00b


Joined: 05 Aug 2005
Posts: 44
Location: San Luis Obispo, CA

PostPosted: Sun Sep 03, 2006 2:44 am    Post subject: Reply with quote

Hi,

I performed all the steps in the instructions before running the script, and everything worked great! Now I'm trying to run your script, but it quits quickly with an error:

Code:

MATT-LAPTOP mstanisz # ./recompile-entire-system-20060901
./recompile-entire-system-20060901 - Recompile Entire System Helper

Script version as of $Date: 2006-09-01T14:15:49.548823Z $

Written in 2006 by Guenther Brunthaler

This script will generate another script to be run by you. That other script
will then recompile each and every package in the whole system in the correct
order.

This will typically be required on a major GCC upgrade.

IMPORTANT: Do not execute this script before all of the following prerequisites
are met:

* Portage tree is up-to-date (emerge --sync)

* Your system is up-to-date (emerge --ask --update --deep --newuse world)

* gentoolkit is available. (The script uses it.) If you are unsure, just do an
'emerge --update gentoolkit' and it will be emerged unless it is already
installed.

* The new compiler you want is already *installed*. (No packages need have to be
recompiled with it yet. It also need not be the currently selected default
compiler version yet.) As GCC allows multislot installations, it is not a
problem in Gentoo to have both your current and a new compiler be installed at
the same time.

* If all the above conditions are met, and no more packages need to be compiled
in order to have an up-to-date system, set the new compiler as the new system
default compiler using "gcc-config".

* If you want to change your Gentoo system profile to a new one using
eselect profile, it is now also the right time to do it.

* Only then continue running this script!

Press [Ctrl]+[C] now in order to abort processing if some of the above
preconditions are not met. Come back and re-run this script after you have
managed to establish all the preconditions as specified.

Press [Enter] now to continue if you dare.

Collecting list of packages and evaluating installation order...
Died at ./recompile-entire-system-20060901 line 450.
MATT-LAPTOP mstanisz #


Could you help me? Thanks!

Matt
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Sep 03, 2006 5:14 pm    Post subject: Reply with quote

KD-120RD wrote:
I have a question here: What is the purpose of switching the profile?


A Gentoo profile sets lower and upper margins for the versions of some important tools (in the list of installed "system" packages), such as the GCC.

Packages which require a compiler (or other system tool) outside of the version range defined by your current Gentoo profile will be masked. That is, "emerge --update" will skip such versions. And "emerge --search" or "esearch" (if installed) will not show such versions to you.

KD-120RD wrote:
I thought gentoo is a meta-distribution. Just update your packages and configs (ie. baselayout) and there you are.


This is correct. But the catch is that updates will just not be performed for packages which are outside the scope of your current profile... ;-)

But what are profiles good for at all?

Profiles protect users from changes that are so dramatic, that their entire system (or at least large portions of it) would have to be recompiled.

One such a change would be an upgrade from GCC 3 to GCC 4.

Without profiles, as soon as GCC4 has been marked as "stable", all Gentoo users running "emerge --update" would find themselves recompiling their whole system, which takes days if not weeks to complete.

Certainly, not all users would be happy then...

In order to avoid this, there are profiles: They allow the users to decide themselves when it is the right time to take such a big step and update their profile to the latest one. (And my guide might be very helpful in this case.)

KD-120RD wrote:
Besides I'm using gentoo for about two years now and as I recall I have not manually swicthed my profile.


You are not alone! I am also still using the 2006.0 profile. I actually used my guide to update from GCC 3.3 to GCC 3.4 - and not to GCC 4. But it's all the same game.

KD-120RD wrote:
When I take a look at my /etc/make.profile it is linked against ../usr/portage/profiles/default-linux/x86/2006.0/
Is that possible?


It is perfectly possible.

In fact, nearly everyone who has not yet updated to this newest 2006.1 profile will see the same.

Note: You can update your profile by using "eselect profile".
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Sep 03, 2006 5:43 pm    Post subject: Reply with quote

grimm26 wrote:
i don't understand the significance of my system profile matching my kernel. It's just default use flags and stuff, right?


The dependency between the system profile and your kernel is the linux-headers package.

This package contains a copy of selected header files which are taken from some linux kernel version (at the time the respective linux-headers package was assembled).

The header files in the linux-headers package are then used as the basis on which your glibc will be built.

The most important information which glibc gets from linux-headers is the syscall numbers.

Have you ever wondered how glibc, being a user-mode library, can actually output a character on your console - which typically belongs to the system, not to you (your user account)?

Glibc implements its write() function by stuffing the write() arguments into processor registers, loading the syscall-number for "write" into another register, and then performing an "int 0x80" processor instruction.

This instruction transfers control to the Linux kernel, which will verify the parameters and perform the write operation if your process is allowed to do so.

And the "syscall"-Number comes from one of the header files in linux-headers.

Now you can also see the relation between your current kernel version and glibc: If the version of the linux-headers package and your actual kernel drift too far apart, it may be that your new kernel provides syscall numbers that are not yet part of your (older) linux-headers package.

Admittedly, syscall numbers nearly never change, and new syscall numbers are also added only very rarely.

But *sometimes* they change, and then the linux-headers (and thus glibc) must be updated in order to take advantage of this.

Besides this, there are also different things in linux-headers which might be of interest for application programs (via glibc): IOCTL numbers. As more and more devices are supported by the Linux kernel, more and more IOCTL numbers (and related structs) are also added to the linux-headers.

And applications which want to use such IOCTLs might thus require newer versions of the linux-headers package.

On the other hand, it certainly would be a problem if linux-headers referred to a newer version of the Linux kernel than is actually installed on the machine.

In this case, the header files would contain IOCTLs that are not yet supported by your current kernel.

So, profiles are again the solution: The profile defines the *oldest* linux kernel that must be available, and linux-headers will have copies from the header files of that kernel.

As the profile ensures that your actual kernel will be at least as current as the headers in the linux-headers package, those header files will never contain anything your kernel does not support.


Last edited by Guenther Brunthaler on Tue Sep 19, 2006 9:48 pm; edited 1 time in total
Back to top
View user's profile Send private message
Guenther Brunthaler
Apprentice
Apprentice


Joined: 22 Jul 2006
Posts: 214
Location: Vienna

PostPosted: Sun Sep 03, 2006 5:59 pm    Post subject: Reply with quote

CPUguy387 wrote:

Code:

Collecting list of packages and evaluating installation order...
Died at ./recompile-entire-system-20060901 line 450.



Hmmm - this is very strange: Line 450 closes a the pipe which was used for reading the packages... For READING!

I could understand if there was an error OPENING it - but closing it?!

CPUguy387 wrote:
Could you help me? Thanks!


I'll try my best.

As I am unable to figure out the cause of the problem, you need to provide me with more information.

It may be easiest to perform some of the actions in the script manually, so we can find out which of the steps does not work.

First see whether the following commands work:

# emerge -pe system > system.list
# emerge -pe world > world.list

When running these commands, they should not print any error messages.

Then, do a

# less system.list

and after that a

# less world.list

and see if both files contain a list of packages, or whether there are obvious problems in the contents of those files.

My script assumes both files are non-empty contain lists of packages.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 1 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum