HOWTO Build binary packages for every package on your system

[Kream]

HOWTO Build binary packages for every package on your system

Thanks to ferringb on #gentoo-portage and assorted folks on #linuxhelp and #perl for this.

If you have systems which use compatible CHOST and march/mcpu settings, then you can easily build binary packages on one system and share them on the others. I do this a lot, so i needed to easily make binaries for all the packages on my system after a little ill advised meddling with distcleaner cleaned out my /usr/portage/packages/All/ directory, containing 1.2GB of precious binary files.

Before starting, I recommend that your system be in as consistent a state as possible. Run cleanup/update scripts like

Code: Select all

/usr/portage/dev-lang/perl/files/perl-cleaner

and

Code: Select all

revdep-rebuild

and if you like, prune all unneeded packages by doing

Code: Select all

emerge -P

READ THE WARNING BEFORE YOU DO THIS!

What we're going to do is create a list of all packages installed in your system and pass that to quickpkg.

Go to /var/db/pkg (which contains the ebuilds and ebuild metadata for the _installed_ packages on your system.)

Code: Select all

cd /var/db/pkg

Generate a listing of all packages in your system and output it to the file /root/installed.

Code: Select all

find . -type d -mindepth 2 -maxdepth 3| sed -e 's:^\./::' > ~/installed
cd /root

For neatness' sake, sort the output

Code: Select all

sort installed > sortedinstalled 

Since quickpkg will accept requests for packages with versions only when it's an atom, i.e, when it is preceded with a =, we need to prefix each line with a = so that each line reads =xfce-base/xffm-4.2.0 rather than xfce-base/xffm-4.2.0.

Code: Select all

sed -i -e 's:^:=:' sortedinstalled

Now instead of having each package atom on a separate line, we we need them all in a single line, separated by a space:

Code: Select all

cat sortedinstalled | perl -pe 'local $/;($_=<>) =~ s/\n/ /g; ' > needed_packages 

now edit this file and right at the beginning, put quickpkg, so that it looks like this:

Code: Select all

vim needed_packages

Code: Select all

quickpkg =app-admin/gentoo-rsync-mirror-1.0-r4 =app-admin/gkrellm-2.2.5 (truncated)

now run the file and you'll get the packages in /usr/portage/packages/All

Code: Select all

bhim root # sh ./needed_packages 

* Building package for gentoo-rsync-mirror-1.0-r4... [ ok ]
* Building package for gkrellm-2.2.5... [ ok ]
* Building package for gnome-system-tools-1.0.2-r1...

Don't celebrate yet!
Before continuing, we need to understand a few things. The Gentoo .tbz2 format contains package metadata which tells portage many things about the package, such as which part of the tree the package belongs to. As a certified portage oldtimer, I've seen the tree grow from 12000 packages to 120,000 and have seen many packages move in the portage tree. An example is fftw, the fast Fourier transform library. This moved from dev-libs/fftw to sci-libs/fftw. Because portage stores tree metadata like this in the package, while portage might require fftw-3.0.1 to be installed, it won't accept the binary package you're offering it for the simple reason that the metadata says that it's actually dev-libs/fftw which you have rather than sci-libs/fftw.

This situation will cause you to go insane - while portage on the system you're trying to build binaries for will happily accept the vast majority of packages, it will cryptically refuse to accept binaires for some package. As the maintainer of a series of media labs in settlement colonies in New Delhi, this caused me no end of problems.

The purported solution to your woes is fixpackages. This reads the binary files that you have and updates the metadata that's present to match the information stored in
/usr/portage/profiles/updates/[1-4]Q-200*, quarterly update information for all binary packages.

Fixpackages has no documentation. Fixpackages does not tell the user clearly what it does [1]. Fixpackages is written in python, so it's opaque to losers like me who don't know python (or any other programming language for that matter). Fixpackages take a long time to run. Fixpackages is absolutely ESSENTIAL if you want to build binary packages for other machines and retain your sanity at the same time.

So run fixpackages.

Code: Select all

fixpackages 

Performing Global Updates: /usr/portage/profiles/updates/2Q-2002
(Could take a couple of minutes if you have a lot of binary packages.)
.='update pass' *='binary update' @='/var/db move'
s='/var/db SLOT move' S='binary SLOT move' p='update /etc/portage/package.*'
................ (truncated)

Once it's done, you'll have a set of packages in /usr/portage/packages/All that are CONSISTENT with the current portage tree.

Here's a consolidated listing of the code for your convenience:

Code: Select all

cd /var/db/pkg
find . -type d -mindepth 2 -maxdepth 3| sed -e 's:^\./::' > ~/installed
cd /root
sort installed > sortedinstalled 
sed -i -e 's:^:=:' sortedinstalled
cat sortedinstalled | perl -pe 'local $/;($_=<>) =~ s/\n/ /g; ' > needed_packages 

edit:

Code: Select all

vim needed_packages

Code: Select all

quickpkg =app-admin/gentoo-rsync-mirror-1.0-r4 =app-admin/gkrellm-2.2.5 (truncated)

Code: Select all

bhim root # sh ./needed_packages 

cheers,
Aniruddha "Karim" Shankar

[1] http://forums.gentoo.org/viewtopic.php?t=55863

[Voltago]

I don't want to curb your enthusiasm, but for building binaries of all installed packages I do

Code: Select all

qpkg -I -nc | grep -v dev-java | xargs quickpkg

(java binaries are not worth building, since compilation is quick).
The tips for ensuring consistency first are surely helpful, though.

[Kream]

wow. sometimes I overlook the easy ones.

thanks

and i'm a big BG2 fan myself as well

[jamesm]

I don't want to curb your enthusiasm, but for building binaries of all installed packages I do

Code: Select all

qpkg -I -nc | grep -v dev-java | xargs quickpkg

(java binaries are not worth building, since compilation is quick).
The tips for ensuring consistency first are surely helpful, though.

I've been trying to build binary packages for everything installed on a system, but your code example no longer works since qpkg is deprecated.

Instead I used,

Code: Select all

/home/jamesm $ cd /var/db/pkg
/var/db/pkg $ find -mindepth 2 -type d -exec quickpkg '{}' \;

YMMV

[yoshi314]

qpkg -I -nc does not work for me,

you can use eix -cI to list all installed packages in short format. all it takes is to do some cut magic to remove irrelevant data from the output and you're set.

(i believe it's also faster using eix, but i couldn't check, since qpkg refuses to work )

[avx]

I'm doing it like this:

Code: Select all

eix -I --format-compact '<category>/<name>' | grep -v Found | xargs quickpkg

Works like a charm for me

[deluge]

and how about:

Code: Select all

quickpkg `qlist -IC | tr '\n' ' '`

works great for me