[NEWS] Final step to smaller, better Portage tree

dol-sen · Posted: Tue Feb 05, 2008 11:17 pm Post subject:

For all those interested in the porthole --[bad/missing] digest problem. It has been fixed in cvs for a long, long long time now. Unfortunately I had not had the time to finish some relatively important feature enhancements & subsequent bugs. I do believe I have everything important fixed now and will be releasing porthole-0.6.0 very soon. I just have to update the install files and ebuild.

see this thread for more about porthole.

Thanks.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

brankob · Apprentice Joined: 29 Apr 2004 Posts: 188

I never undesrstood the need for digest as a separate entity anyway.

What is it for ? To protect ebuild from third party tampering ? If so, what should protect digest itself from tampering ?

Why can't each ebuild simply be "signed" at the end so that for each "good"ebuild certain hash function gives same result ?

This way, one wouldn't have to have separate file for the digest ( and so spare those almost 4K and extra filessek).... :roll:

mottmar

brankob · Apprentice Joined: 29 Apr 2004 Posts: 188

Yes, but digest in-ebuild could adress that, too.

Once you know that you can trust that ebuild, you can use it to check package files...

jlh · Posted: Wed Feb 06, 2008 1:50 pm Post subject:

If you question the separate digest files, you could also question a number of other things about the layout of the portage tree. But it all boils down to the problem that almost all files in the tree are very small, which is highly inefficient unless you mount it on a separate filesystem with smaller blocks; but even then, you're still stuck with like 100'000 files which is still awfully slow to handle.

I was thinking about just having a zip files for every package, which would contain the ebuilds for all versions, the metadata, digest, change log, and patches. This would help with the small-file problem at the cost of having to fix all tools that need to read that tree and at the cost of that it's more difficult to manually access the files. But I bet 10'000 files would sync much faster than 100'000 files.

But then I suddenly wondered why my system has to store and regularly update the exact build instructions and patches for various versions of 10'000 packages that I will probably never ever care about. Talk about a waste of time! Of course, my system should know about which packages exist, so that I can search for them, but that doesn't mean it needs to store and update the entire ebuild and patches for of them. So I was wondering if it would be possible to replace the entire portage tree with a list (database, files, whatever) of available packages in each available version, with its name, description, dependencies and of course digests, just enough information so that I can query that list for available packages and build the dependency tree without net access. But the ebuilds themselves would only be downloaded from a mirror when I really ask it to install the package; and at that time you need net access anyway in order to download the tarball. Such a list of packages would still contain enough informations to do most tasks offline, but would be much much smaller and faster to sync, since it won't contain patches and build instructions.

But maybe this idea is flawed somehow... comments?

brankob · Apprentice Joined: 29 Apr 2004 Posts: 188

my thoughts exactly.

Also, instead of having one ebuild for each version of the package one could have one file, which could have several parts, possibly with first part done as conventional ebuild and other parts diffed relative to first part for subsequent versions or somesuch.

I think that with multiversion ebuilds and without the requirement for client to actually rsync whole tree for each update things could be much lighter and snappier.

Also, it would be fine to have decent default provision for emerging binary, precompiled packages, if possible without the need to have exact same gcc and options as the prebuilt packages.

brankob · Apprentice Joined: 29 Apr 2004 Posts: 188

Oh, and it would be nice to have the tool to prune distfiles map, sine after some time it accumulates many unneeded packages.

Same goes for packages map ( when keeping the source of built packages )

Just at the moment I saw original post about smaller portage tree I did du -sh /usr/portage and got something over 18GB !

After emptying distfiles and packages and redownloading all disfiles again final number is around 3.7 GB with distfiles for some 970 packages...

jlh · Posted: Wed Feb 06, 2008 3:21 pm Post subject:

BlackEdder · Posted: Wed Feb 06, 2008 3:27 pm Post subject:

mikegpitt · Advocate Joined: 22 May 2004 Posts: 3224

If I use a few overlays with the digest files stored the old way, will things get automatically updated to the new digests when I run the 1st emerge, or is it backwards compatible?