Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to recover a lost world file
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
fonic
n00b
n00b


Joined: 20 Mar 2011
Posts: 4

PostPosted: Sun Mar 20, 2011 10:19 am    Post subject: How to recover a lost world file Reply with quote

Hi everybody,
over the years, the Gentoo forum helped me out quite a few time and therefore I think it's time to give something back. So here's my first post - hopefully it proves to be useful for others. If I chose the wrong category for this post, please move it to where it belongs.


How your world file may get lost
I guess there are countless ways of how to loose the portage world file. In my case, two simple but serious mistakes lead to an empty world file. As many others, I'm using rpm in Gentoo for the sole purpose of installing drivers for printers and scanners made by Brother (e.g. the MFC series). Yesterday, I wanted to update those drivers. Therefore, I issued rpm -e brscan[...] in order to remove the old drivers prior to installing the new ones.

This was the first serious mistake, since you must NEVER ever use rpm without the --nodeps option if rpm is not the main package manager on your system. Anyway, issuing said command left me with a completely messed up system: several important files like 'libstdc++.so' and 'stdio.h' were missing, gcc was unable to create executables and so on. I wasn't able to emerge anything, basic programs like eix and locate stopped working - long story short, the system was almost unusable. However, I was able to use links and wget to download a current stage3 tarball which I planned to use to get the basic system functions back online.

I knew that extracting the tarball would probably overwrite a lot of important files, so I created a backup of /etc. That was the second serious mistake I made - before extracting a stage3 tarball in order to recover a broken system, one should backup EVERYTHING, not only /etc. So, after extracting the tarball most parts of the system like portage, gcc and actually most other applications as well were back online, but the portage world file located at '/var/lib/portage/world' had been overwritten and was now empty.

All of this took place after 2am. As it is mentioned in How I met your mother, "No good things happen after 2am. After 2am, just go to bed". They were totally right... Gentooish high five...


Recovering a lost world file
There is that well-known command called regenworld, which attempts to recover lost world files. In my case, that did not help at all, since regenworld tries to find out about package installations that took place in the past by parsing portage's log files. Those were gone too by extracting the tarball, therefore regenworld did not report a single package on my system.

After trying several administrative commands, I noticed there are at least two ways of getting a list of currently installed packages: equery and eix. Both of them use their own databases and seem to be unaffected by a missing world file. Sadly, the package lists provided by equery and eix contain ALL packages currently installed on the system, i.e. programs, libraries, fonts, codecs etc. The world file, however, should only contain packages that were installed on demand.
Code:
# equery list -i
# eix-installed -a

But then I came up with an idea: packages that are installed on demand most likely have no dependencies to other packages - if they had, they would have already been pulled in by other packages during the emerge process. I verified that thought with a few equery depends <package> on packages like vlc, amarok, kdebase-meta etc. which I knew had been installed on demand. The results were very promising.


Using a script to automate the recovery process
Based on the idea described above I created a small but effective recovery script which collects a list of all currently installed packages and then automatically checks the dependencies for each package. The script creates a text file containing all packages without dependencies, which may directly be used as a replacement for the lost world file. On my system, the script recovered over 90% of my world file (80 entries out of 85 total). The recovery process takes some time (about 30 min. in my case) since equery depends queries are very slow - if you have an idea on how to speed things up, feel free to comment accordingly.

Some packages won't be reported by the script because of their special dependency structure. One example is 'sun-jre-bin', which depends on 'virtual/jre' and is therefore omitted by the script. But those few missing packages may easily be spotted by checking the result of emerge -pv --depclean after copying the recovered world file to '/var/lib/portage/world'. In my case, I only had to add five packages reported by depclean in order to have my world file recovered to 100%.

All in all, my system was up and running again after about three hours (including the time it took to write the script) which is well below the amount of time it would have taken to install Gentoo and all my applications from scratch. By using the script, other users with a similar problem might be able to recover their system in less than an hour, I guess.

Find the recovery script I created attached below. Feel free to comment if you have any suggestions on how to improve it.

Best,
fonic


Last edited by fonic on Sun Mar 20, 2011 11:43 am; edited 4 times in total
Back to top
View user's profile Send private message
fonic
n00b
n00b


Joined: 20 Mar 2011
Posts: 4

PostPosted: Sun Mar 20, 2011 10:35 am    Post subject: Recovery script Reply with quote

Since there seems to be no way of attaching files to posts, here's the script using the code tag:

Code:
#!/bin/bash

#
# recover_world.sh by fonic
# Created: 03/19/11
# Modified: 03/19/11
#

# Configuration
dwork=./rebuild_world
fstep1="1_world_raw"
fstep2="2_world_stripped"
fstep3="3_world_recovered"

# Create working directory and flush files
echo -n "Creating working directory... "
mkdir -p $dwork
cd $dwork
> $fstep1
> $fstep2
> $fstep3
echo "ok"


# Step 1:
# Get a raw list of the currently installed packages. Both eix-installed
# and equery will provide such lists. Note that either list will include
# all sorts of packages (i.e. libraries, fonts, codecs etc.), not only the
# main programs we're looking for.
echo -n "Grabbing a raw list of currently installed packages... "
#eix-installed -a > $fstep1
equery -q list -i > $fstep1
numpackages=$(cat $fstep1 | wc -l)
echo "$numpackages packages found"


# Step 2:
# Process raw list and remove version numbers. The code also accounts for
# packages with additional release information (e.g. qt-webkit-4.6.3-r1).
echo -n "Stripping version information from package list... "
while read package; do
   echo ${package%-[0-9]*} >> $fstep2
done < $fstep1
echo "ok"


# Step 3:
# Process package list and determine the number of dependencies for each
# package. Packages WITHOUT dependencies are most likely packages that were
# installed on demand (and therefore need to be put in the world file).
while read package; do

   # Determine number of depedencies of the current package
   echo -n "Processing \"$package\"... "
   numdeps=$(equery -q depends $package | wc -l)
   if [ $numdeps -eq 1 ]; then
      echo "$numdeps dependency"
   else
      echo "$numdeps dependencies"
   fi

   # If there are no dependencies, a world entry is found
   if [ $numdeps -eq 0 ]; then
      echo "-> Packages was added to the list of recovered world entries."
      echo $package >> $fstep3
   fi

done < $fstep2

# Done, show results and depclean notice
numpackages=$(cat $fstep3 | wc -l)
echo -e "\nThe recovery process is completed. $numpackages world entries were identified and stored in '$fstep3'.\n"
echo -e "Make sure to check 'emerge -pv --depclean' for packages that were not found by this script.\n"
Back to top
View user's profile Send private message
nenemsis5
n00b
n00b


Joined: 23 Oct 2007
Posts: 69

PostPosted: Sun Mar 20, 2011 4:48 pm    Post subject: Reply with quote

does the command: "regenworld" not doing the same ??
Back to top
View user's profile Send private message
dwbowyer
Apprentice
Apprentice


Joined: 18 Apr 2008
Posts: 154

PostPosted: Tue Mar 22, 2011 6:12 am    Post subject: Reply with quote

Quote:
does the command: "regenworld" not doing the same ??


For those that didn't actually bother to read the entire original post:

Quote:

Recovering a lost world file
There is that well-known command called regenworld, which attempts to recover lost world files. In my case, that did not help at all, since regenworld tries to find out about package installations that took place in the past by parsing portage's log files. Those were gone too by extracting the tarball, therefore regenworld did not report a single package on my system.


Now, I do wonder that extracting the tarball over a live system would have overwritten log files though... is that the default portage /var/log/build.log rather than the individual logs written to /var/log/portage/<package-group>:<package-name>-<packageversion><datetime>.log?
Back to top
View user's profile Send private message
fonic
n00b
n00b


Joined: 20 Mar 2011
Posts: 4

PostPosted: Tue Mar 22, 2011 9:50 pm    Post subject: Reply with quote

dwbowyer wrote:

For those that didn't actually bother to read the entire original post:

Thank you! I can't stand people commenting on something they actually haven't read...

Quote:

Now, I do wonder that extracting the tarball over a live system would have overwritten log files though... is that the default portage /var/log/build.log rather than the individual logs written to /var/log/portage/<package-group>:<package-name>-<packageversion><datetime>.log?

Well, I can't tell exactly what happened or how it happened. My focus that night was to get the system running again, therefore I didn't spend much time on finding out what lead the data loss. It might have also been rpm that caused the missing portage log files since it destroyed a lot on the system. Fact is, regenworld could not find anything in that situation while my way (or: my script) did. And even if regenworld helps in 90% of those cases, I think it never hurts to have a backup solution.
Back to top
View user's profile Send private message
Nicias
Guru
Guru


Joined: 06 Dec 2005
Posts: 387

PostPosted: Tue May 03, 2011 6:28 pm    Post subject: Reply with quote

I trimmed your script, and just used the last stage to clean up my world file. I saw I could delete 25 of the 99 entries from my world file.
Then I uncovered a problem with it. If a package has a conditional dependency, it will flag it as a dependency. For instance I have gimp installed from world, and gtkam installed. gtkam has the following line in its RDEPEND "gimp? ( >=media-gfx/gimp-2 )" Since I don't have the gimp use flag set, it doesn't acually depend on gimp, but equery sees that it does. So your script flags gimp as non-world-worthy.
Back to top
View user's profile Send private message
Nicias
Guru
Guru


Joined: 06 Dec 2005
Posts: 387

PostPosted: Tue May 03, 2011 9:21 pm    Post subject: Reply with quote

After doing more work, I'd be very cautious about using this. I had 99 packages in my world file, it only said I needed 74. Upon further inspection, I actually need 92. So it missed almost 25% of the packages that should be in my world.

Having said that, it's still better than nothing, if you have no choice.

I would just suggest backing up.
Back to top
View user's profile Send private message
drutten
Tux's lil' helper
Tux's lil' helper


Joined: 31 Dec 2003
Posts: 103
Location: Umeå, Sweden

PostPosted: Mon Oct 03, 2011 5:44 pm    Post subject: Reply with quote

Thank you for this post and script! Did the same thing (unpacked stage3 tarball) one evening when everything was going wrong. Should have gone to bed instead. I figured a solution like this should work and thought of writing a script, but then realised it must already exist somewhere :lol:


Unfortunately they changed the use-pattern of equery -q list

Code:
# equery -q list -i
List installed packages matching the query pattern

!!! Default action for this module has changed in Gentoolkit 0.3.
!!! Use globbing to simulate the old behavior (see man equery).
!!! Use '*' to check all installed packages.
!!! Use 'foo-bar/*' to filter by category.

Usage: list [options] pkgspec

options
 -h, --help              display this help message
 -d, --duplicates        list only installed duplicate packages
 -b, --missing-binpkgs   list only installed packages without a corresponding binary package
 -f, --full-regex        query is a regular expression
 -m, --mask-reason       include reason for package mask
 -I, --exclude-installed exclude installed packages from output
 -o, --overlay-tree      list packages in overlays
 -p, --portage-tree      list packages in the main portage tree
 -F, --format=TMPL       specify a custom output format
              TMPL       a format template using (see man page):
                         $location, $mask, $mask2, $cp, $cpv, $category, $name, $version, $revision, $fullversion, $slot, $repo, $keywords


You should either uncomment the eix-installed command and comment out equery, or change it to
Code:
equery -q list '*' > $fstep1


Thanks again!
/André
Back to top
View user's profile Send private message
soltoo
n00b
n00b


Joined: 02 Jul 2011
Posts: 15

PostPosted: Thu Nov 03, 2011 8:56 pm    Post subject: Reply with quote

Thanks for the script.

I did notice that dev-lang/python was not in my world file before and this script didn't include it either.

Running "emerge -p system | grep python" doesn't show it. Running "emerge -p world | grep python" doesn't show it either.

Plus dev-lang/python was out-of-date until I ran "emerge python" even after I ran "emerge world".

Admittedly I'm running Gentoo Prefix on Solaris 10. I started using it in July 2011--I don't think I ever had a python 2.6 installed.

Code:
eselect python list
Available Python interpreters:
  [1]   python2.7 *

nothing else.

Does anyone else see this behavior too?
Back to top
View user's profile Send private message
mittah
n00b
n00b


Joined: 30 Oct 2012
Posts: 2

PostPosted: Tue Oct 30, 2012 1:06 pm    Post subject: Reply with quote

Very good and useful script! Thank you!

My using stats:
found by script: 124
missed: 16
Nice result!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum