[advice about portage]portage waste lots of diskspace

[xiaosuo]

today i typed:

[seank]

[Zarhan]

[jbc28]

Just downloading an ebuild list won't work without some alterations as the dependencies of an ebuild are potentially dependent on its USE flag settings.
Downloading a list of ebuilds without any further info (no dependency information) might be better: portage could then just fetch the ebuilds as needed, though it would still need to know the status of each package (x86, ~x86, -x86 etc). The problem then would be if you're left without net access but have access to precompiled binaries as I think the *.tbz files need the relevant ebuilds. Perhaps the solution to this would be to add the appropriate ebuild into the tbz, that way it works as a stand alone.

I'm tempted to agree that portage does use a lot of disk space but more importantly uses perhaps more bandwidth than is necessary. Maybe addition
of a 'fetch-ebuild' flag could fetch ebuilds as needed?

J

[Torangan]

If you'd like to save bandwith, you'd probably better suggest a feature to use patches instead of full downloads whenever possible. That way portage could save a really great amount of bandwith. I think it should be possible to offer patches against the previous version which must reside in distfiles and then create a new tarball out of the previous version and the patch.

[xiaosuo]

i am sorry for forgeting the directory "/usr/portage/distfiles"
but you can see

[oberyno]

Rsync_excludes works for me.

[xiaosuo]

i use ext3. using reserfs4 is a good idea, but not the ultimate method.
if we change the portage work function. We just need to download the ebuilds's KEYWORDS SLOT USE DEPENDS ...., and we can design a good data struct for these ebuilds infomation, and use lzo(quick compress) compress this files, and we can download this instead of sync with the portage.
we do not need the portage cache because the lzo file is also the cache. this method can speed up searching and depending.
With so much benefit, WHY NOT change ?
_________________
I love freedom
and I want to fly
so I love linux
because I love linux
I choose gentoo

[Zarhan]

If the diskspace issue is such a big deal, then just share the /usr/portage directory over NFS (or any other network filesystem). Then you only need to host the files once for all your local computers.

Support for diffs (ie. when you have kernel 2.6.9-sources and upgrade to 2.6.10, portage downloads only the patchset, not the entire source package) will probably be in soon (there are already multiple approaches, search these forums). That should solve some of the bandwidth usage issues.

Also, if you want a compressed version, just use portage snapshots (get the tarball). Emerge-webrsync does just what you propose, it downloads a "snapshot" tarball. Maybe you can access it directly via squashfs or something instead of decompressing it, but it's there.

This is the way it has been since forever and by "forever" I'm also including FreeBSD ports tree. Granted, FreeBSD offers binary packages for far more packages and they get updated more often than 4 times a year, but still, it's a good system.

The balance on which parts to host on client (your) computer and which on the network has been decided long ago and I doubt it will change. There are guidelines, such as what kinds of files you can include in /usr/portage/<category>/<package>/files directory (small patches can be handily stored there) or when you should just put an URL into the ebuild for download. Shifting more of the stuff to the server side - well, you'll suddenly need more powerful servers.

Also, in it's current form portage is really simple, as in elegant. It's a notch up from FreeBSD's ports which is really just a collection of makefiles and handles a few things better (such as SLOTs). Point is, that the network transactions caused by portage (or ports) are really simple: Just wget a file via http from a world of mirrors. No database access, nothing special.

With a (completely) server-based model you would first get a list of packages, choose one, send it back to server, the server would have to calculate your dependencies, send the list back, you would check which ones you already have installed, request ebuilds for the rest, etc. Yes, this can be optimized a bit. Still, the load of the servers would increase - there are much more users than there are servers.

You see, even if you store the dependency info locally (the IUSE, RDEPEND lines) and can perform the dep calculations locally - it would increase complexity and server load. Right now, the only stuff that needs to be downloaded are the STANDARD source tarballs - directly from the software authors with a gazillion of mirrors. If the rest of the ebuild information has to be downloaded separately for each emerge - then you'll either need "gentoo-specific" source tarballs with the information attached - OR centralized ebuild servers. Considering that there are probably like millions of Gentoo users emerging this and that every minute, these servers would be under heavy load constantly. So there would have to be a heavy mirror infrastructure - just for Gentoo - not usable by anyone else. Rsync is much better - you just sync to it like once a week - and you can even host your own local rsync mirror for your own network.

Considering all these possibilities to squeeze portage to a smaller footprint (hosting over NFS, rsync_excludes for those games-*-categories you don't want, cleaning up your distfiles, possibility to use compressing filesystems), I think that this will not change in foreseeable future.

[tcbounce]

Hi,

I have 4 computers on a LAN where I live and only one of them has a portage tree I do the NFS mount things as well, but with one benefit.

If you run SELinux when writing to NFS shares (after making a package) portage will die. The fix to this is to use the experimental NFS patch and NFS tools, then your security permissions are done automatically over the network.

I have downloaded some modifications to portage that clean out the old distfiles, if the file is not current for the architectures (ARCH_KEYWORDS) I use.

Finally the /usr/portage /usr/portage/distfiles and /usr/portage/packages are not the only thing you should be sharing!

I share my /etc/portage and my portage overlays over NFS, so I can have heaps of custom ebuilds and don't have to tweak my portage settings on any other computers. It's like software installation profiles under M$, but better!

If the machines are seperated by some distance, instead of NFS you should setup a portage rsync mirror for your tree and other portage files (overlays & conf), and use portage's PKG_BINHOST feature to install the packages via FTP.

There is a nice howto about how to create a portage binary mirror. I'll post it to the forum in about 10 mins when I find it, about how to automatically clean out distfiles and packages. (or at least packages - there are other scripts for distfiles).

[ttuttle]

[Skinkie]

[Gentree]

[tuam]

[Skinkie]

[yngwin]

[Naib]

1Gig out of say a HD that is how big?

sure it is quite "expensive" if the HD is 10gig in size but with harddrive cost really low its not to much of an issue
_________________

[Gentree]

yes emerge -p is a bit sluggish but you will spend more time discussing how it could be improved that the sum total of all the seconds it makes you wait.

So dicussing speed is a waste of time and discussing disk usage is a waste of space.

Just about wraps this topic up. Not that the original question was not a fair one, these things should be asked.

Now where's the "stop watching this..."
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86

[yngwin]

[tracker]

[volkris]

[neysx]

[Skinkie]

[Gergan Penkov]

[Skinkie]