Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Major System Upgrades - Post Mortem
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 734
Location: US-FL-EST

PostPosted: Sat Mar 20, 2010 9:24 pm    Post subject: HOWTO: Major System Upgrades - Post Mortem Reply with quote

Pre-Introduction -4) Huh?!?!?

This is the last of a series four posts describing and walking through the complete process of performing a Major Upgrade on a Gentoo based system. The other posts are:

Introduction - Post Mortem

If you've been following along in Post I - Prep, Post II - Initial Upgrades Plus and Post III - Emerge World, you'll know all of the steps needed for a major Gentoo Linux upgrade and all installed packages. But that's not the end of the story. There are always little 'gotchas' that creep in and try to spoil all the hard work you did.

In addition to the inevitable and usually unique 'gotchas' there are a host of little niggles to remember as well.

In no particular order, these, then, are the 'gotchas' and 'niggles' that occurred on the example laptop:

emerge @preserved-rebuild

Since the example laptop runs the unstable branch of portage, @sets are a supported function. The theory here is that the one or more packages might get 'broken' if a library they depend on changes. What's happening is that the called library which was just upgraded changes it's version number but the package which uses it still points to the old version number. While the calling package is will happily accept working with the new library version, it needs to be told to do so.

For the stable version of portage, 'revdep-rebuild' is the mechanism for doing this. For the unstable version of portage, 'emerge @preserved-rebuild' is the mechanism used. However, in the unstable version of portage, you can still use revdep-rebuild as your fallback.

Under usual circumstances where regular "emerge --sync && emerge -uND world" happens regularly, a single pass of either 'emerge @preserved-rebuild' or 'revdep-rebuild' is sufficient to fix mismatched dependent libraries. Under the circumstances of a major upgrade on a system which hasn't seen an upgrade for a long time, 'sufficient' may not be enough.

On the example laptop, I executed:
Code:
# emerge @preserved-rebuild

There were a total of 69 packages selected for re-emerging. At the completion of that process, I received messages indicating I needed to run 'emerge @preserved-rebuild' again!! I executed (again):
Code:
# emerge @preserved-rebuild

This time, 4 packages were selected for re-emerging. All completed successfully. However, I got exactly the same 'emerge @preserved-rebuild' message again for exactly the same libraries.

Lather, Rinse. Repeat. -- Same messages again.

revdep-rebuild

Trick: If you manually delete the old libraries indicated in the @preserved-rebuild {libraries preceded with a 'hyphen' symbol}, you can guarantee forcing the rebuilding of the needed packages through 'revdep-rebuild'.

I deleted the indicated deprecated libraries and executed:
Code:
# revdep-rebuild


This time, revdep-rebuild reported 7 packages needed to be re-emerged {versus 4 for @preserved-rebuild}. Then revdep-rebuild failed on the very first package selected telling me there were no ebuilds available for the indicated package.

The package it was complaining about was 'qca'. The problem here was that I didn't indicated to revdep-rebuild emerge the latest versions of the packages to be rebuilt. The then current 'qca' packages was version 1.6. Version 2.02 is the only version available. On the other hand, 'emerge -uND world' never picked up 'qca' for upgrading. That {rarely} happens sometimes. Ohhh-Kayyyy. I executed:
Code:
# emerge qca

The emerge ran successfully. I kicked off again:
Code:
# revdep-rebuild

Once again, revdep-rebuild failed. This time it wanted to emerge 'PyQt:0'. Whoa! Nelly! There is no longer 'PyQt'. I did say this was an old system after all. But not that old. Indeed, since I had upgraded to kde-4.1.3 previously, PyQt had been upgraded at that time as well. Instead, there is only PyQt4. After shaking my head, I execute:
Code:
# emerge PyQt4"

This emerge completed successfully.

While the above emerge was running, it occurred to me that perhaps revdep-rebuild might be trying to tell me something else. I inspected the revdep-rebuild report of broken libraries more closely. This time, I realized that revdep-rebuild was trying to re-emerge packages which were dependent on python-2.5. At some previous time, when python was upgraded from 2.5 to 2.6, about 20 python-2.5 library files were left behind from the emerge -C python-2.5 process. No. I haven't a clue as to how this happened and I don't think it has anything to do with pertinent gentoo devs. I guess that at some previous update, there must have been a problem I didn't notice.

I executed:
Code:
# rm -rf /lib/python/2.5
# revdep-rebuild

This time, only 2 packages were selected for re-emerging and they completed successfully. Yeah! :D

etc-update

After any upgrade with a fair number of packages, there will be changes to various configuration files located in a variety of places. The original update tool for all these configuration files is 'etc-update'.

For the example laptop, there were 163 configuration files waiting to be updated. My general rule of thumb is to select and inspect the updates of all configuration files in /etc/conf.d. There are also a {very} few files outside of /etc/conf.d which need manual inspection. One generally {and sometimes painfully} learns which of these are important to the user over time.

For the example laptop, there were 8 configuration files in /etc/conf.d. There was also /etc/samba/smb.conf. After manually selecting, visually confirming whether to update or abandon the update these few files, I select '-5' and allowed all the other files to autoupdate. YMMV. Viewer discretion is advised!

Note that a number of configuration files are not just subject to option changes, but these few are subject to serious functional changes. Depending on age of system, watch out for configuration files having to do with:
  • udev
  • lmsensors <== /etc/sensors.conf changed to /etc/sensors3.conf
  • /etc/conf.d/net <== major changes here
  • /etc/samba.conf <== moved to /etc/samba/smb.conf
  • /etc/hostname {domanname, etc} <== completely different!
  • timezone <== completely different!
  • /etc/X11/xorg.conf <== in theory, can be deleted for most people. X will usually pick the most optimal choices for you. Seems to work.

There are others as well. These just immediately come to the top of my mind.

python-updater

As part of the pre-upgrade cleanups, we made use of 'lafilefixer' and also cleaned up all existing installs of kde. Now we're going to do some post upgrade cleanup using python-updater. Besides, we already know there was unexpected python-2.5 cruft earlier so it's especially important to run python-updater. Execute:
Code:
# python-updater -p

I've chosen '-p'retend because I want to see what python-updater finds before it tries to fix anything.
Code:
* Active version of Python 3:     3.1
 *   Adding to list: app-emulation/vmware-workstation:0
 *   Adding to list: app-office/openoffice-bin:0
 *   Adding to list: app-pda/libopensync:0
 *   Adding to list: dev-java/antlr:0
 *   Adding to list: dev-java/java-config:0
 *   Adding to list: dev-java/javatoolkit:0
 *   Adding to list: dev-libs/boost:1.42
 *     check: manual [Added to list manually, see CHECKS in manpage for more information.]
 *   Adding to list: dev-python/PyQt:0
 *   Adding to list: dev-python/numeric:0
 *   Adding to list: dev-python/pyopengl:0
 *   Adding to list: dev-python/pyrex:0
 *   Adding to list: dev-python/python-fchksum:0
 *   Adding to list: dev-python/pyxf86config:0
 *   Adding to list: dev-python/pyxml:0
 *   Adding to list: media-libs/pdflib:5
 *   Adding to list: sys-libs/libieee1284:0
 *   Adding to list: x11-libs/vte:0
 *     check: manual [Added to list manually, see CHECKS in manpage for more information.]
 * emerge -Dv1 --keep-going -p app-emulation/vmware-workstation:0 app-office/openoffice-bin:0 app-pda/libopensync:0 dev-java/antlr:0 dev-java/java-config:0 dev-java/javatoolkit:0 dev-libs/boost:1.42 dev-python/PyQt:0 dev-python/numeric:0 dev-python/pyopengl:0 dev-python/pyrex:0 dev-python/python-fchksum:0 dev-python/pyxf86config:0 dev-python/pyxml:0 media-libs/pdflib:5 sys-libs/libieee1284:0 x11-libs/vte:0

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds to satisfy "dev-python/PyQt:0".

Dang it! I thought we had resolved PyQt earlier. But wait ...

Trick: Copy the emerge command line but this time, add the -u option. This is a perfectly valid thing to do because, after all, we've been doing a major system upgrade. Execute:
Code:
# emerge -uDv1 --keep-going -p app-emulation/vmware-workstation:0 app-office/openoffice-bin:0 app-pda/libopensync:0 dev-java/antlr:0 dev-java/java-config:0 dev-java/javatoolkit:0 dev-libs/boost:1.42 dev-python/PyQt:0 dev-python/numeric:0 dev-python/pyopengl:0 dev-python/pyrex:0 dev-python/python-fchksum:0 dev-python/pyxf86config:0 dev-python/pyxml:0 media-libs/pdflib:5 sys-libs/libieee1284:0 x11-libs/vte:0

Yes!!
Code:
These are the packages that would be merged, in order:

Calculating dependencies... done!

!!! All ebuilds that could satisfy "x11-libs/qt:3" have been masked.
!!! One of the following masked packages is required to complete your request:
- x11-libs/qt-3.3.8b-r2 (masked by: package.mask)
/usr/portage/profiles/package.mask:
# Ben de Groot <yngwin@gentoo.org> (01 Mar 2010)
# Grand mask of qt:3 and remaining reverse dependencies
# pending removal on 21 Mar 2010 (bug 283429)

- x11-libs/qt-3.3.8b-r1 (masked by: package.mask)

(dependency required by "dev-python/PyQt-3.17.6" [installed])
(dependency required by "dev-python/PyQt:0" [argument])

What I thought I knew I didn't know. What I had forgotten was that qt3 was required with earlier versions of qt4 in order to provide qt3 support when and where as needed. The new version of qt4 no longer requires this. Execute:
Code:
# emerge -Cp PyQt

We want to confirm what's going on and that we no longer need PyQt-3.X
Code:
>>> These are the packages that would be unmerged:                                                                                                                                         
                                                                                                                                                                                           
 dev-python/PyQt
    selected: 3.17.6
   protected: none
     omitted: none

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Confirmed - now execute:
Code:
# emerge -C PyQt

Copy the python-updater generated emerge command again, add the '-u' option and remove PyQt from the list of packages to be re-emerged. Execute:
Code:
# emerge -uDv1 --keep-going -p app-emulation/vmware-workstation:0 app-office/openoffice-bin:0 app-pda/libopensync:0 dev-java/antlr:0 dev-java/java-config:0 dev-java/javatoolkit:0 dev-libs/boost:1.42 dev-python/numeric:0 dev-python/pyopengl:0 dev-python/pyrex:0 dev-python/python-fchksum:0 dev-python/pyxf86config:0 dev-python/pyxml:0 media-libs/pdflib:5 sys-libs/libieee1284:0 x11-libs/vte:0

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U ] dev-python/pyrex-0.9.8.6 [0.9.8.5] USE="-examples" 241 kB
[ebuild     U ] gnome-base/libgtop-2.28.0 [2.24.4] USE="-debug" 0 kB
[ebuild     U ] dev-python/pyopengl-3.0.1_beta2 [3.0.0_beta1] USE="-tk (-doc%)" 681 kB
[ebuild     U ] app-pda/libopensync-0.39 [0.36] USE="python -doc -test% (-debug%)" 0 kB
[ebuild     U ] dev-cpp/libgnomecanvasmm-2.26.0 [2.23.1] USE="-debug -doc -examples" 0 kB

Total: 5 packages (5 upgrades), Size of downloads: 922 kB

All good! Re-run the emerge command again and this time remove the '-p' option. Allow to complete.

I've included both my erroneous conclusion above and the entire sequence of steps here to demonstrate a few points:
  1. What you think you know is not always what is real.
  2. It's important to be observant as to process messages. Portage really does try to let you know what's what.
  3. Major upgrades consist of many small and not so small otherwise independent upgrades. Each of these package upgrades have the potential for special handling.
  4. Before and after cleanups are important!


rc-update show/add

First, execute:
Code:
# rc-update add xdm

Remember, we deleted xdm from the default rc. Before we forget, we need to put it back.

Now is also a good time to review available services. I usually execute:
Code:
# rc-update show | sort -b -d
# ls /etc/init.d

I want to see is a dictionary style sort of default services that currently start upon boot and what services are available. So I do an 'rc-update show', pipe the results to 'sort' and tell sort to strip leading blanks and sort in dictionary order. I don't use the extended listing option of 'ls' because I'm hoping that the displayed results will all fit on one screen.

Tip: If you're using grub, add vga=794 to the kernel command line to have the kernel display more lines and columns on your screen. This is suitable for screens capable of 1280x1024 resolution. If you reboot and your screen is blank and it is capable of 1280x1024 resolution, then your screen does not properly report it's capabilities back to the kernel when queried. In that case, you can reboot and try vga=791 {1024x768). If you do not understand how to use the line editor mode in grub, then do not try this tip.
Code:
                acpid |      battery                                           default         
            alsasound |      battery                                           default         
             bootmisc | boot                                                                   
          consolefont | boot                                                                   
                cupsd |      battery                                           default         
                 dbus |      battery                                           default         
                devfs |                                         sysinit                       
        device-mapper | boot                                                                   
                dmesg |                                         sysinit                       
                 fsck | boot                                                                   
                 hald |      battery                                           default         
               hdparm |      battery                                           default         
             hostname | boot                                                                   
              hotplug |      battery                                           default         
              hwclock | boot                                                                   
              keymaps | boot                                                                   
            killprocs |                                                                shutdown
                local |      battery nonetwork                                 default         
           localmount | boot                                                                   
              modules | boot                                                                   
             mount-ro |                                                                shutdown
                 mtab | boot                                                                   
               net.lo | boot                                                                   
           ntp-client |                                                        default         
               procfs | boot                                                                   
                 root | boot                                                                   
               rsyncd |      battery                                           default         
            savecache |                                                                shutdown
               serial | boot                                                                   
                 sshd |      battery                                           default         
                 swap | boot                                                                   
               sysctl | boot                                                                   
            syslog-ng |      battery                                           default         
         termencoding | boot                                                                   
                 udev |                                         sysinit                       
       udev-postmount |                                                        default         
              uptimed |      battery                                                           
              urandom | boot                                                                   
           vixie-cron |      battery                                           default

...

NetworkManager  dhcdbd        hald         mDNSResponderPosix  net.wlan0   pydoc-3.1    serial        udev
acpid           dhcpcd        hdparm       mdnsd               netmount    reboot.sh    shutdown.sh   udev-dev-tarball
alsasound       dhcpd         hostname     mit-krb5kadmind     network     root         slpd          udev-mount
apache2         dhcrelay      hotplug      mit-krb5kdc         nfs         rpc.gssd     snmpd         udev-postmount
bluetooth       dmcrypt       hwclock      modules             nfsmount    rpc.idmapd   snmptrapd     uptimed
bootmisc        dmesg         iplog        mount-ro            nscd        rpc.pipefs   sshd          urandom
consolefont     dmeventd      keymaps      mtab                ntp-client  rpc.statd    staticroute   vixie-cron
consolekit      dnsextd       killprocs    mysql               ntpd        rpc.svcgssd  svnserve      vmware
cpufrequtils    esound        laptop_mode  mysqlmanager        numlock     rpcbind      swap          wpa_supplicant
crypto-loop     fancontrol    linux-logo   nas                 nxserver    rsyncd       swclock       xdm
cupsd           fsck          lm_sensors   net.eth0            pciparm     samba        sysctl        xdm-setup
dbus            functions.sh  local        net.eth1            procfs      saslauthd    sysfs         xinetd
devfs           fuse          localmount   net.lo              pwcheck     savecache    syslog-ng
device-mapper   gpm           lvm          net.lo.openrc.bak   pydoc-2.6   sensord      termencoding

Services not currently started at boot but might be of interest would be things like
  • 'dmcrypt' - I like my privacy and am thinking of segregating data/files in order to maintain desired privacy.
  • 'fuse' - I'm set up for fuse but currently don't have an application that requires it
  • lm_sensors - previous kernels didn't have drivers to support the fan controller chip in the example laptop - currently researching this
  • laptop-mode - this is new to me - perhaps I can get rid of the custom 'battery' runlevel?
  • mysql - add this! I plan on setting up amarok on the example laptop
  • nfs - add this! For access to home network server which houses /usr/portage/distfiles for all the PCs on my network
  • 'numlock' - don't add this! DOH!! the laptop doesn't have a separate number keypad
  • vmware - don't add. Only rarely using the virtual PCs on an irregular basis.
  • 'xdm' well - DOH!


Odds and Ends
Or for those of you from the British Empire "Odds and Sods".

This is a catch all for all the rest of the remaining niggles. Since I re-installed vmware-workstation, It's a good idea to not to forget to configure it:
Code:
# emerge --config vmware-workstation

There are a few kernel settings I see I missed earlier. In particular, I wanted to try out the newly added I2C drivers which are supposed to support the fan controller chip in this lap top. I've updated the kernel and copied 'bzImage' to '/boot/2.6.33-bzImage'. When I reboot, I'll need to re-emerge all the packages which build against the active kernel. For the example laptop, I'll execute:
Code:
# emerge nvidia-drivers vmware-modules

Before starting KDE-4.4.1 for the first time, I made the decision to not migrate the any of my KDE-3.5 or old KDE-4.X settings. Instead, I execute:
Code:
# mkdir ~/KDE.oldsettings
#mv ~/.kde* ~/KDE.oldsettings

There are a number of reasons why I chose to do this. They are:
  • I don't want broken .kde information to confuse the new version of kde. Since I have old 3.5.10, 4.1.3 and 4.2.0 settings information, this is not a trivial concern.
  • I definitely don't want to be involved in attempting to migrate amarok-1.6 embedded mysql data to the new external mysql-5.1 database as used by amarok 2.3.0
  • I'm interesting in seeing all new setting methods etc., so this is a good learning opportunity.

Instead, I'll deal with the KDE new login wizard and spend some time playing with and configuring the desktop to my satisfaction.

This is not a trivial exercise for me. As it happens, I provide 'technical support' to both my mother and also a dear friend. Both have pretty serious vision problems. In fact, my primary reason for being such a hardcore active user of KDE is because KDE lets me do more to natively support the vision impaired than any other windows manager. Gnome need not apply. Really.

Tip 1: For the vision impaired, through real testing, I've found the Veranda font to be the easiest one to read both on screen and in print.

Tip 2: For the vision impaired, depending on the type of impairment, white text on dark backgrounds is usually more legible. Consider this when configuring window colour schemes.

I'll be adding more example little things to not forget as they occur.

In the meantime, I hope you enjoyed the Post Mortem.
_________________
People whom think M$ is mediocre, don't know the half of it.


Last edited by dufeu on Tue Apr 13, 2010 1:29 am; edited 1 time in total
Back to top
View user's profile Send private message
elko
n00b
n00b


Joined: 02 Feb 2010
Posts: 38

PostPosted: Mon Apr 12, 2010 8:03 pm    Post subject: Reply with quote

Hi, first of all, thank you for the guide. Good work.

I believe you have a typo in one command. There should be

Code:

rc-update show | sort -b -d


instead of

Code:

rc-update show | -b -d
Back to top
View user's profile Send private message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 734
Location: US-FL-EST

PostPosted: Tue Apr 13, 2010 1:28 am    Post subject: Reply with quote

elko wrote:
Hi, first of all, thank you for the guide. Good work.

I believe you have a typo in one command. There should be

Code:

rc-update show | sort -b -d


instead of

Code:

rc-update show | -b -d

DOH! You are correct. Thank you!

:D I've always found self editing hard to do.
_________________
People whom think M$ is mediocre, don't know the half of it.
Back to top
View user's profile Send private message
Lars
Tux's lil' helper
Tux's lil' helper


Joined: 06 Feb 2003
Posts: 145
Location: Germany, near baltic sea

PostPosted: Tue Sep 07, 2010 6:44 am    Post subject: Reply with quote

What about:

  • perl-cleaner
  • java-check-environment
  • emerge --metadata and or emerge --regen


Never the less, very nice HOWTO, thanks.
_________________
Quote:
Alles was nicht einfach ist, ist entweder falsch oder zu kompliziert.

V.Glazounov
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum