Portage with p2p

Message

nmcsween · Post by **nmcsween** » Wed May 31, 2006 2:37 am

This has all been said before... Why not change the way you think why should compiler flags even be touched by the average user? Why not distribute binaries that have been compiled with simply -02 and -i686 and create deltas of sse and 3dnow or even profiled feedback?

Matteo Azzali · Post by **Matteo Azzali** » Wed May 31, 2006 9:06 am

rafo wrote: But think for a moment about a Gentoo "emerge". It is open source, yet we use it. By following sound operating procedures we can be confident that we don't run a hacked version of "emerge". The same reasoning goes for an install program that you download from some other organization: There are ways to be sure that what you downloaded is the program that they provide, and not a hacked version.

Emerge has nothing to do with p2p, it seems to me that any rsync or distfile server is identified and well-known , so server admin have some kind
of "responsibility" on what they do (and you can track back they if it's needed), if they would crack users machines, their server could be "removed" from
the list.

The package-building machinery of the Bonus Binary Foundry should be automated and demand driven. If many users have a certain flag combination then the BBF will be hit by many requests for binary packages with that flag combination, and so those packages will tend to be produced early on. Packages for less common flag combinations will also be produced, but it will take longer. There is no need to try and predict which flag combination is the most typical; after a while it is however possible to gather some interesting statistics.

Yes, but in this way don't you tend to "convince" users to use the already built combinations? I mean, X or DE emersion can be quite long,
will not users decide to use all the same flags (and that's quite bad for testing...)

Matteo Azzali · Post by **Matteo Azzali** » Wed May 31, 2006 9:16 am

nmcsween wrote:This has all been said before... Why not change the way you think why should compiler flags even be touched by the average user? Why not distribute binaries that have been compiled with simply -02 and -i686 and create deltas of sse and 3dnow or even profiled feedback?

IMHO you are trying to move from gentoo to another distro: some of these just stick with -O2 -i386 without any delta....
Why not try directly another distro, then?

Hypnos · Post by **Hypnos** » Wed May 31, 2006 9:28 am

My major beef with binary distros were ABI dependency headaches and having to reconfigure binary packages specs when I needed a new option.

The two key objections to having an extensive binary infrastructure in Gentoo seem to be security and USE flags.

Regarding USE flags, every hardware host has a set of default USE flags defined /usr/portage/profiles -- why not just use those as the default? If a user wants to deviate, he can build from source. This would give the best of both worlds between source- and binary-based.

Regarding security, isn't Portage moving to GPG signatures anyway? If there is a central, trusted binary repository, they can authenticate binary packages just like vendors authenticate source packages.

However, I don't know what to do about ABI hell. Even if you fix gcc, glibc, etc. versions, USE flags and CFLAGS can alter compatbility (AFAIK). This is a general Linux annoyance, more than Gentoo's fault, but the problem remains ...

rafo · Post by **rafo** » Wed May 31, 2006 9:44 am

Hi Matteo,

As I see it these kind of "binary" initiatives are just adding choice to Gentoo: You can trade trust, optimization and customization for installation speed. Other friendly initiatives that are also driven by the sometimes perceived need for speedy installation are Kororaa (http://kororaa.org/), Chinstrap (http://chinstrap.alternating.net/), and RR4 Linux (http://www.lxnaydesign.net/). Gentoo takes no responsibility for them, but they are there if you trust them and find them useful. One thing they have in common is to underline that the end result is a valid Gentoo installation: you can tweak flags and re-emerge just like Gentoo promises.

I just helped a colleague with a Kororaa installation. My judgment is that he will benefit from being able to "emerge" this and that, and that he would not have tolerated me fiddling with his machine for several days. Maybe he will take further steps into Gentoo land, maybe not. Maybe in the end he will find that Gentoo is too demanding and go for a commercial boxed distribution. But even if that happens he has already learned a little about what Gentoo is.

rafo · Post by **rafo** » Wed May 31, 2006 9:53 am

Hi Hypnos,

just tossing in a link that explains ABI a little (I have not stumbled over the term before): http://en.wikipedia.org/wiki/Applicatio ... _interface

rafo · Post by **rafo** » Sat Jun 03, 2006 12:14 pm

So far I have been thinking of a p2p scheme for sharing trustable binaries. However, there could really be different distribution mechanisms, suitable for different contexts.

Suppose that you want to deploy Gentoo within an organization (a company, a campus, a hospital, ..). Then it may in fact be cost effective to keep produced binaries on a central server, or a network of servers. These servers would of course be trusted within the organization. Fingerprints could still be checked "just in case".

When a host is being installed or updated it will send requests for binaries (produced under specified preconditions) to the server. If the binary is already available it is of course downloaded. If it is not yet produced the requesting host may handle the "no" response in two ways: Either fall back to local compilation and installation, or hang on to the request until the server has produced the binary and download it then. The latter mode may be implemented in different ways, perhaps a polling scheme is sufficient.

Such a scheme could work very well for, say, a school. Schools are often low-budget operations and may be reusing "outdated" and hence slow hardware. For the administrator of school computers it would be a pain to wait for every box to compile its own software. With a binaries server the first box would take some time to install (still better off than classical Gentoo since binaries are produced on a faster central server), and subsequent boxes would be install rapidly using binaries from the central server.

I am aware that putting binaries on a central server is nothing new--it can be done today, and is maybe done too. The nice thing is that it could be one mode of operation of the "bonus binaries" scheme; whereas p2p sharing could be another. What they have in common is the demand-driven "binaries factory" that allows payback on the investment in compile time while at the same time preserving the full flexibility of Gentoo.

barrymac · Post by **barrymac** » Sat Jul 28, 2007 5:18 pm

How about a p2p portage system so no-one on the planet ever compiles something that's been compiled before by another gentooist?

I'm sure it wouldn't be the most difficult system to set up with bittorrent and some clever way to name torrents after their architectures and use flag options.

Probably the most popular stuff should be permanently seeded.

nixnut · Post by **nixnut** » Sat Jul 28, 2007 5:26 pm

merged above post here.

sugar · Post by **sugar** » Sun Aug 05, 2007 4:38 am

It would save on having to depend on mirrors.

sirdilznik · Post by **sirdilznik** » Sun Aug 05, 2007 4:41 am

Maybe because then it would force someone to also have a torrent client. I think minimum requirements is a good thing for portage. Also which client would you choose as default? I guess it could be added with a USE flag.

Just my thoughts.

sugar · Post by **sugar** » Sun Aug 05, 2007 4:44 am

sirdilznik wrote:Maybe because then it would force someone to also have a torrent client. I think minimum requirements is a good thing for portage. Also which client would you choose as default? I guess it could be added with a USE flag.

Just my thoughts.

libtorrent, and rtorrent if necessary, and you'd start portage as a service I suppose, so it would act as a client.

Sadako · Post by **Sadako** » Sun Aug 05, 2007 6:55 am

It would not save on depending on mirrors, it would just lighten the load on them.
You would still have to ensure that there are seeds for each tarball available at all times.

It would also involve setting up a dedicated tracker, and you would need to publish a new torrent for every single version bump or new ebuild (although I'm sure this could be automated).

Another point is that the majority of files/tarballs on the mirrors are quite small, and in such cases only making them available as torrents simply wouldn't make sense.

There is also the issue of the amount of work it would take to add it to portage.

It's not a terrible idea, and I'd be surprised if it wasnt at least discussed before, but overall I don't think it would really be worthwhile implementing.

codergeek42 · Post by **codergeek42** » Sun Aug 05, 2007 7:26 am

Moved from Off the Wall to Gentoo Chat.

djdunn · Post by **djdunn** » Sun Aug 05, 2007 9:22 am

I think that certain files say over 50 megs a torrent could be an option, much like using FTP is an option.

I'm sure some of the mirrors would be able to handle seeding also,

for example when new versions of KDE or OOo or whatnot come out having a torrent as an option would probably help lessen the demand on the mirrors.

but again i don't think that every single file would benefit from being a torrent but some could benefit greatly.

Telexen · Post by **Telexen** » Sun Aug 05, 2007 9:57 am

I don't like it. There are things torrents are good for and there are things they're not good for.

You're not going to lighten the load on portage servers that much with torrents - even from big ones. And I'll be able to download the large files I need MUCH faster with the side effect of a little extra server load than I would with torrents.

I'm not sure why people think torrents are the answer to everything. I've downloaded my share linux distros and such with torrents, and have never seen a single download max out my 3mbit cable connection - yet I always do downloading large files from a portage server.

nixnut · Post by **nixnut** » Sun Aug 05, 2007 10:43 am

merged above seven posts here.

sugar · Post by **sugar** » Mon Aug 06, 2007 4:01 am

nixnut wrote:merged above seven posts here.

Thanks nixnut. I didn't realise that this seems to come up regularly. I should browse Gentoo Chat more often eh.

nwmcsween · Post by **nwmcsween** » Wed Aug 08, 2007 12:17 am

For something that can be attained without a large overhaul why not just have solid releases such as kernel 2.6.22 and create deltas for any of the updates to said kernel? KDE does it SUSE does it.

Corona688 · Post by **Corona688** » Wed Aug 08, 2007 5:27 am

nwmcsween wrote:For something that can be attained without a large overhaul why not just have solid releases such as kernel 2.6.22 and create deltas for any of the updates to said kernel? KDE does it SUSE does it.

That could get really complicated as more and more patches get added on, patching the patched patch of the patch; they already get dozens as is. Suse doesn't have to maintain more than one kernel, either.

It'd save in downloading kernel images if that could be conquered, but really, how many kernel tarballs do users download? Might there be better savings to be had improving something else? Maybye those ginormous KDE tarballs could be diffed.

nwmcsween · Post by **nwmcsween** » Wed Aug 08, 2007 6:40 am

Corona688 wrote:
nwmcsween wrote:For something that can be attained without a large overhaul why not just have solid releases such as kernel 2.6.22 and create deltas for any of the updates to said kernel? KDE does it SUSE does it.
That could get really complicated as more and more patches get added on, patching the patched patch of the patch; they already get dozens as is. Suse doesn't have to maintain more than one kernel, either.

It'd save in downloading kernel images if that could be conquered, but really, how many kernel tarballs do users download? Might there be better savings to be had improving something else? Maybye those ginormous KDE tarballs could be diffed.

You don't patch a patch of a patch you patch the base with the update not patch the patch with the patch. Simply have a base and patch it every new release . e.g xorg 1.1.1.1 -> xorg 1.1.1.1-rc1-patch then xorg 1.1.1.1 -> xorg 1.1.1.1-rc2-patch simple.

Roman_Gruber · Post by **Roman_Gruber** » Sat Aug 11, 2007 12:15 am

ever heard of sabayon linux?

minervaix · Post by **minervaix** » Sat Sep 15, 2007 6:43 am

After recently installing gentoo on one of my boxes using all the packages from another gentoo machine via emerge -k $package with my PKGDIR set, and installing a full system with xorg and kde in a couple of hours. I decided all those gentoo users out there with buildpkg in their make.conf should set up some P2P software so we can all make use of pre built packages, especially good for the i need it NOW!! situations, we can call it the GENTELLA network. whatda think?

omp · Post by **omp** » Sat Sep 15, 2007 7:07 am

minervaix wrote:whatda think?

That it would be easy to inject malicious code into the source prior to building something.

bunder · Post by **bunder** » Sat Sep 15, 2007 7:10 am

Merged the above two threads.

Portage with p2p

P2P portage anyone?

Why not enable portage to download tarballs as torrents?

gentella network

Re: gentella network