Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO Build binary packages for every package on your system
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Kream
n00b
n00b


Joined: 19 Apr 2002
Posts: 30
Location: New Delhi, India

PostPosted: Thu Mar 31, 2005 8:11 am    Post subject: HOWTO Build binary packages for every package on your system Reply with quote

HOWTO Build binary packages for every package on your system

Thanks to ferringb on #gentoo-portage and assorted folks on #linuxhelp and #perl for this.

If you have systems which use compatible CHOST and march/mcpu settings, then you can easily build binary packages on one system and share them on the others. I do this a lot, so i needed to easily make binaries for all the packages on my system after a little ill advised meddling with distcleaner cleaned out my /usr/portage/packages/All/ directory, containing 1.2GB of precious binary files.

Before starting, I recommend that your system be in as consistent a state as possible. Run cleanup/update scripts like
Code:
/usr/portage/dev-lang/perl/files/perl-cleaner

and
Code:
revdep-rebuild


and if you like, prune all unneeded packages by doing
Code:
emerge -P
READ THE WARNING BEFORE YOU DO THIS!

What we're going to do is create a list of all packages installed in your system and pass that to quickpkg.

Go to /var/db/pkg (which contains the ebuilds and ebuild metadata for the _installed_ packages on your system.)

Code:
cd /var/db/pkg


Generate a listing of all packages in your system and output it to the file /root/installed.

Code:
find . -type d -mindepth 2 -maxdepth 3| sed -e 's:^\./::' > ~/installed
cd /root


For neatness' sake, sort the output
Code:

sort installed > sortedinstalled


Since quickpkg will accept requests for packages with versions only when it's an atom, i.e, when it is preceded with a =, we need to prefix each line with a = so that each line reads =xfce-base/xffm-4.2.0 rather than xfce-base/xffm-4.2.0.

Code:
sed -i -e 's:^:=:' sortedinstalled


Now instead of having each package atom on a separate line, we we need them all in a single line, separated by a space:

Code:
cat sortedinstalled | perl -pe 'local $/;($_=<>) =~ s/\n/ /g; ' > needed_packages


now edit this file and right at the beginning, put quickpkg, so that it looks like this:
Code:

vim needed_packages


Code:
quickpkg =app-admin/gentoo-rsync-mirror-1.0-r4 =app-admin/gkrellm-2.2.5 (truncated)

now run the file and you'll get the packages in /usr/portage/packages/All
Code:

bhim root # sh ./needed_packages

* Building package for gentoo-rsync-mirror-1.0-r4... [ ok ]
* Building package for gkrellm-2.2.5... [ ok ]
* Building package for gnome-system-tools-1.0.2-r1...

Don't celebrate yet!
Before continuing, we need to understand a few things. The Gentoo .tbz2 format contains package metadata which tells portage many things about the package, such as which part of the tree the package belongs to. As a certified portage oldtimer, I've seen the tree grow from 12000 packages to 120,000 and have seen many packages move in the portage tree. An example is fftw, the fast Fourier transform library. This moved from dev-libs/fftw to sci-libs/fftw. Because portage stores tree metadata like this in the package, while portage might require fftw-3.0.1 to be installed, it won't accept the binary package you're offering it for the simple reason that the metadata says that it's actually dev-libs/fftw which you have rather than sci-libs/fftw.

This situation will cause you to go insane - while portage on the system you're trying to build binaries for will happily accept the vast majority of packages, it will cryptically refuse to accept binaires for some package. As the maintainer of a series of media labs in settlement colonies in New Delhi, this caused me no end of problems.

The purported solution to your woes is fixpackages. This reads the binary files that you have and updates the metadata that's present to match the information stored in
/usr/portage/profiles/updates/[1-4]Q-200*, quarterly update information for all binary packages.

Fixpackages has no documentation. Fixpackages does not tell the user clearly what it does [1]. Fixpackages is written in python, so it's opaque to losers like me who don't know python (or any other programming language for that matter). Fixpackages take a long time to run. Fixpackages is absolutely ESSENTIAL if you want to build binary packages for other machines and retain your sanity at the same time.

So run fixpackages.

Code:
fixpackages


Performing Global Updates: /usr/portage/profiles/updates/2Q-2002
(Could take a couple of minutes if you have a lot of binary packages.)
.='update pass' *='binary update' @='/var/db move'
s='/var/db SLOT move' S='binary SLOT move' p='update /etc/portage/package.*'
................ (truncated)

Once it's done, you'll have a set of packages in /usr/portage/packages/All that are CONSISTENT with the current portage tree.

Here's a consolidated listing of the code for your convenience:
Code:

cd /var/db/pkg
find . -type d -mindepth 2 -maxdepth 3| sed -e 's:^\./::' > ~/installed
cd /root
sort installed > sortedinstalled
sed -i -e 's:^:=:' sortedinstalled
cat sortedinstalled | perl -pe 'local $/;($_=<>) =~ s/\n/ /g; ' > needed_packages

edit:
Code:

vim needed_packages
Code:

quickpkg =app-admin/gentoo-rsync-mirror-1.0-r4 =app-admin/gkrellm-2.2.5 (truncated)


Code:

bhim root # sh ./needed_packages

cheers,
Aniruddha "Karim" Shankar
:D
[1] https://forums.gentoo.org/viewtopic.php?t=55863
_________________
https://forums.gentoo.org/search.php?search_id=unanswered
Back to top
View user's profile Send private message
Voltago
Advocate
Advocate


Joined: 02 Sep 2003
Posts: 2539
Location: In the city of dreaming spires

PostPosted: Thu Mar 31, 2005 11:01 am    Post subject: Reply with quote

I don't want to curb your enthusiasm, but for building binaries of all installed packages I do
Code:
qpkg -I -nc | grep -v dev-java | xargs quickpkg

(java binaries are not worth building, since compilation is quick).
The tips for ensuring consistency first are surely helpful, though.
Back to top
View user's profile Send private message
Kream
n00b
n00b


Joined: 19 Apr 2002
Posts: 30
Location: New Delhi, India

PostPosted: Fri Apr 01, 2005 9:35 am    Post subject: Reply with quote

:oops: :oops: :lol: :lol: :lol: :lol: :roll: :roll: :roll:

wow. sometimes I overlook the easy ones.

thanks :)

and i'm a big BG2 fan myself as well
_________________
https://forums.gentoo.org/search.php?search_id=unanswered
Back to top
View user's profile Send private message
jamesm
n00b
n00b


Joined: 13 Jun 2003
Posts: 22

PostPosted: Mon Aug 07, 2006 6:54 am    Post subject: Reply with quote

Voltago wrote:
I don't want to curb your enthusiasm, but for building binaries of all installed packages I do
Code:
qpkg -I -nc | grep -v dev-java | xargs quickpkg

(java binaries are not worth building, since compilation is quick).
The tips for ensuring consistency first are surely helpful, though.


I've been trying to build binary packages for everything installed on a system, but your code example no longer works since qpkg is deprecated.

Instead I used,

Code:
/home/jamesm $ cd /var/db/pkg
/var/db/pkg $ find -mindepth 2 -type d -exec quickpkg '{}' \;


YMMV :)
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 817
Location: PL

PostPosted: Mon Aug 07, 2006 6:39 pm    Post subject: Reply with quote

qpkg -I -nc does not work for me,

you can use eix -cI to list all installed packages in short format. all it takes is to do some cut magic to remove irrelevant data from the output and you're set.

(i believe it's also faster using eix, but i couldn't check, since qpkg refuses to work )
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2070

PostPosted: Thu Aug 10, 2006 7:04 pm    Post subject: Reply with quote

I'm doing it like this:

Code:
eix -I --format-compact '<category>/<name>' | grep -v Found | xargs quickpkg


Works like a charm for me :)
Back to top
View user's profile Send private message
deluge
Apprentice
Apprentice


Joined: 28 Jan 2004
Posts: 157

PostPosted: Wed Aug 16, 2006 12:52 pm    Post subject: Reply with quote

and how about:

Code:
quickpkg `qlist -IC | tr '\n' ' '`


works great for me :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum