Portage with p2p

[nephros]

See also: several previous discussions.

Personally I switched to using the deltup way because it does get the job done of minimizing bandwidth use, and also it is there and working
_________________
Please put [SOLVED] in your topic if you are a moron.

[ssmaxss]

I think that the more important thing is to get deltup server working stable, fast and produce normal deltas rather than redirecting to delta that "404 Not found". It will decrease mirrors load too. Mirrors could even have only one version of each package and only deltas to newer/older version, but this will need adding such information to ebuilds (or implementing some logic to find shortest path of deltas to needed file)

[frew]

Hey all, I was thinking about this today because my campus recently instated a 3 gig bandwidth limit and currently has no Gentoo mirror. I was thinking it would be great if gentoo had a p2p thing that could be set up anywhere, so on Campus I tell all the gentoo people to use my p2p server blah blah and we all share inside the network. So, I was just thinking that this thing that we are hopefully going to really persue would have the following attributes:

1: you can easily make your own server
2: you can easily use a different server
3: you can easily modify various stuff about the server (only networked connections or something)

Obviously I haven't taken networks yet, but I thought that this would be cool.

[ssmaxss]

See http://gentoo-wiki.com/HOWTO_Use_dijjer_for_portage . I think it could help you. Maybe we all should setup it? Please post your comments about it if you try it!

[CalPaterson]

Bittorrent was initially designed to distribute GNU/Linux operating systems. I seem to recall that Mandrake was the original test. 1/1000th of the traffic they currently have would be pretty nice for the gentoo servers. BT is a good choice.

There are also some nice clients for it; rtorrent is my favourite. One of the things you really need to consider about using distributed networks is the relatively vast amounts of cpu they consume (in comparision to http/ftp). I suppose gentoo users generally have more powerful computers, but running a java or python client can cause a problem for people.

[ssmaxss]

Bittorrent is good for distributing larege files. But for distributing one or two distfiles it will be bad. The only way to use bittorrent is to create one torrent for distfiles directory and recreate it on every change. But this torrent will be quite large so syncing will be slow. I think protocols like DC or Gnutella2 will be good choice. Gentoo can have its own DirectConnect hub that will share only distfiles and maybe portage.

[rafo]

Yesterday I was upgrading packages on one of my Gentoo machines, so I had some time to think

It has been said so many times before: Wouldn't it be nice to have a "binary option" in Gentoo so that you could upgrade packages quickly? And the answers go along the lines "USE flag combinations make it unrealistic", "security is lost", "well then try Ubuntu".

I do agree that a general all-embracing solution does not seem possible. But what if we could come up with a scheme that does not change anything in Gentoo as we know it, but offers trustable binaries as a bonus for those that find it useful?

This could mean that if you run Gentoo on a fairly mainstream machine, and don't care to tweak USE flags a lot, you would get many of your upgrade packages as binaries. When you embark on a tweaking path you would increasingly compile source packages, no worse than today.

Now it can be said that if you want binaries and you don't care to tweak you could just as well run Ubuntu/Debian/SuSE/whatever. I don't agree. Gentoo has many other advantages besides tweakability, for example the beautiful implementation of startscripts and the overall ease of upgrade.

So how could it be done? Of course I don't have a ready-to-launch solution, it is more like I would like to contribute pieces to a puzzle and I surely don't have them all. Here goes,

1. Binary packages should be shared over a p2p network. This means that no central organization has to put up file space for the packages.

2. Security is a must. My initial idea is that there should be a trusted authority that runs a "fingerprint service", allowing every binary package to be checked before it is installed.

3. A source package is identified by its (qualified) name and version. From a particular source package and version it is then possible to create a huge number of different binary packages. Changes in CFLAGS, USE flags, compiler version, etc will cause the resulting binary package to be different. Think of an "environment descriptor" that is a textual encoding of all parameters that affect the resulting binary.

4. From the user perspective, think of a "p2p-emerge" command that operates roughly like this:

A. Ask the fingerprint service for the fingerprint of package P compiled with environment E, where E is an environment descriptor. If there is no fingerprint available, resort to source compilation, producing a binary package. Make the binary package and the associated environment descriptor available to peers.

B. Otherwise send a p2p query for package P compiled with environment E. If there is no reply within reasonable time, resort to source compilation, producing a binary package. Make the binary package and the associated environment descriptor available to peers.

C. Otherwise if the p2p download succeded, compute the fingerprint of the downloaded package and compare it against the trusted fingerprint. If fingerprints don't match, discard the downloaded package and resort to source compilation, producing a binary package. Make the locally produced binary package and the associated environment descriptor available to peers.

D. Otherwise if fingerprints match, install the package and make it (and the environment descriptor) available to peers.

5. The fingerprint service will receive fingerprint requests for various {package, environment} combinations. A reply containing the fingerprint will be sent if the service has one.

If the service does not yet have a fingerprint it starts a package compilation process in the background, setting up the prescribed environment, producing a Gentoo binary package, taking its fingerprint, and storing the fingerprint. This is the only way that a trusted fingerprint can enter into the world. The binary package may then be discarded immediately (however, since it is a valid binary package it may in fact be made available over the p2p sharing network).

The service is free to abort a compilation process for any reason: Outrageous or exotic USE flags, package known to be little demanded, package compile-time is negligible, environment cannot be set up, whatever.

The service is also free to garbage collect stored fingerprints as it sees fit: Low hitrate may be the only criterion that is needed.

That is all I have, at a conceptual level. The scheme will be a bonus to mainstream users, who will frequently get valid binary packages from the p2p network, and decreasingly useful to users with ever more advanced USE flag settings.

Some work is needed to arrive at an environment descriptor format that is sufficiently general and extensible. Some XML format maybe? Note that if we start out with a poor format that doesn't capture all aspects of the compilation environment we will not have any disasters, just poor performance: binary packages will turn out with different fingerprints in enviroments that are in reality different although our descriptor does not capture it.

It is also likely that some packages will be totally ill-behaved for this scheme. Suppose an executable within a package has a function that prints out a line stating "compiled 26 May, 2006 on host kgbvax". Every p2p participant will then produce a binary package that has a differing fingerprint and so it will never match the trusted fingerprint. Actually the fingerprint service could use idle time to recompile packages that it has already compiled, and mark the package as "unsuitable for sharing" if the fingerprint turns out to be not reproducible.

The fingerprint service will benefit from computational power. The scheme allows a high degree of parallelism: background compilations could be handed out to a pool of processors. This is just a performance issue however. Another one is the front-end towards the network, which could implement some kind of load sharing as needed.

The "trusted authority" is bacically any player that is trusted by its users community. If the scheme is successful it may of course be adopted by Gentoo Foundation at some later point.

Oh well. There must be some weak spots. You fire away

[xef]

That would be perfetc...

We could use a binary arch to use binary packages just like debian. For the packages we want to install with some customization (cflags and use flag) use the source arch.

[rafo]

Hi xef,

I am not convinced that a new arch is needed. And even if you set some non-default CFLAGS and USE flags you may benefit from sharing as long as others have made the same settings as you (and provided that the fingerprinting service has succeeded to build a binary package with those settings).

[anello]

P2P (as root???) is a nightmare!!! What if you're looking for a package that isn't that popular. You get stuck with getting it from a 56k modem user (exaggereted) which in the end takes way longer than compiling it from source.

I would also never trust a binary file that isn't approved by an authority.
And the only way to accomplish this is that only a trusted singel person or little group can be held responsible for its package and validate the files to their clear conscience. Like ebuild maintainers or the original source devs right now. There should be binary maintainers that provide binarys for all possible USE Flag combinations for different Processor Types and of course different arch's (loads of work!!!).

Thats the first thoughts that came to my mind. I love the idea of having a binary "distribution" of gentoo (see VidaLinux (no binary packages though/just a stage3 combo), but I think that the implemention is a bigger pain in the ass then the benefit is worth it, since there are loads of binary distributions already out there ready to go.
_________________
Antonino Catinello | http://catinello.eu

[rafo]

Hi anello,

of course if bandwidth is a worse bottleneck than compilation speed then the scheme is no good. But my feeling is that on the average p2p downloading of binary packages should be faster than producing them locally. Maybe the program that performs the download could include some sanity check that aborts a download if it goes very slow, with local compilation as fallback, no worse than today.

Binary packages must be validated and that is also an essential part of the scheme. But producing the trusted fingerprints is intended to be an automated process, driven by users wishing to download particular packages compiled with particular flag settings. Once the scheme gets going there should be no manual work needed.

[anello]

[rafo]

Hi again anello, one important thing: The fingerprints are only created by the trusted authority, which runs a demand driven service that produces binary packages from proper Gentoo source packages, computes their fingerprints, throws away the binary packages and keeps the fingerprints. Ordinary users cannot submit fingerprints.

A user can indeed produce any binary package and offer it over the p2p network. A malicious user may piece together a binary package that contains a trojan, say. It will however be rejected by all downloaders since its fingerprint differs from the fingerprints that the trusted authority has published.

And, if the authorized service does not have the fingerprint for a certain package+flags yet (production of the binary package may be ongoing, or not started) the p2p download will not be attempted at all. As always the fallback is local compilation of the source package.

[mikegpitt]

If you want binary packages then try out Portage BINHOST:
http://gentoo-wiki.com/TIP_Using_PORTAGE_BINHOST

I've never tried it, but it looks like there are server that have many arch's up already.

If someone does/has tried it I'd be curious how you like it...

[rafo]

I didn't know that there are already such servers. Thanks for the pointer!

Three things come to my mind. First of all, security. You can choose to trust the organization that runs the service if you like, just like you can choose to trust any issuer of a LiveCD. From a paranoid viewpoint though, a malicious service operator could use the service for planting malicious software on a great number of hosts.

Second: The flexibility of Gentoo is lost. The binary packages on such servers are bound to use vanilla flag settings, more or less.

Third: Scalability is an issue, in theory at least. If the service becomes popular there will be a substantial traffic load on the servers; this can of course be dealt with but it does not come for free. Traffic is not for free in a p2p scheme either, but the costs will be shared among the users benefitting from it.

[rafo]

It strikes me that two performance improvements are possible.

First, there is the problem with packages that don't have a reproducible fingerprint (see the long article of yesterday, or simply consider a program that has a banner stating when it was compiled). What the fingerprint service should do is to feed the binary package into the p2p system, so that a future request for the package will cause it to be downloaded and usable copies will be spread.

This does not lower security; data flows outwards from the trusted service, not into it.

Once the link between the fingerprint service and the p2p system is there it may just as well be used all the time. Thus as soon as a package is produced and fingerprinted by the trusted service, feed it into the p2p network.

The second thing is that the p2p request for a package should also include the trusted fingerprint (obtained in step 4A). The peer should only allow a download if the fingerprint in the request matches the locally computed fingerprint of the package. In that way it is avoided that the package is downloaded just to be immediately discarded by the requesting side.

Also, if there was a mismatch on the peer side it is most likely that the binary package has a fingerprint that differs from the authorized one. It will therefore fail in every future download attempt, so it should be removed from the p2p space. This does not necessarily mean that the package is incorrect, it may simply be that the fingerprint algorithm fails to produce a repeatable result for that package.

I think I should point out that my competence in the p2p area is rather minimal. I have used Limewire a couple of times, getting a general feel for the idea, that is all. If I write things that don't make sense, let me know! Anyway it seems that this scheme requires a rather flexible and programmable p2p framework, a kind of "p2p Meccano kit". Perhaps there is already such a thing out there?

[tkdfighter]

[mikegpitt]

I dont think security on a centralized host (or distributed for that matter) is that much of a problem, since all you would need is the md5 checksums to verify the build is legit. The source code d/l's are just as vulnerable to attack as binaries, but again md5 saves the day.

[alistair]

I personally think it would be easiier for a binary distro to move towards a source ~arch (or more likely hard-masked) tree rather than gentoo to move "backwards".

Why don't you start up a debain based project that uses debain-ish stable tree and a gentoo-ish development tree. Then ppl could have stable binary packages in a matter of minutes or could compile source from ~arch.

ALL you would have to do is make sure that you have a program to automate the coversion & compilation of a ebuild package to a deb package. oh and also solve how that whole binary/source interoptablility problem (from a package management and dependancy standpoint). Would be interesting to have a package manager go

I'm installing this package
is package to be binarry or source
It needs package1 and package2 as dependancies
package1 is binary and installed. great...
package2 isn't installed. dam do I binary install it or source install it.
source therefore install source package2 then package as source/binary.

I look forward to see this.
_________________
______________
Help the gentoo-java project. Visit Gentoo Java Project

what good are admin powers if you don't abuse them for personal gain - mark_alec

[rafo]

It may well be that something could be done based on packages, or just package formats, borrowed from Debian. But I would rather stay in Gentoo land if possible.

The challenge, as I see it, is to come up with a scheme that meets all these characteristics:

- No new features are required in Gentoo, no administrative or maintenance burden is laid on Gentoo, nor any additional bandwidth or filespace requirements on Gentoo servers.

- Binary packages are produced by an independent and trustable party (working name: Bonus Binary Foundry). Independent in the sense that it does not require support or approval from Gentoo, and trustable in the trivial sense that there are users that trust the binaries and other software it produces.

- Production of binary packages is demand driven, automated, and not limited to just a single set of cflags/useflags/etc. If there is interest in a certain flag setting, as manifested by "bbf-emerge" requests, a binary package may be produced for that setting.

- BBF manages a p2p network for distribution of the binaries. This means that the users benefitting from the scheme also contribute filespace and bandwidth. BBF itself does not have to store all binaries that it produces for a very long time, but it will keep the fingerprints for a long time.

- A binary is installed only if its recomputed fingerprint can be validated against the fingerprint stored at BBF.

- Users who choose to benefit from BBF will find that the "bbf-emerge" command (provided by BBF) will sometimes retrieve and install binaries, sometimes not. There are no guarantees. The fallback is always plain "emerge"and local compilation of the source package.

[Genone]

Congratulations, this is probably one of the best proposals about this topic I've seen so far, though it still has quite a few problems:

• fingerprint: better to use real signatures for verification
  
• build environment: one big proble you will face is to track and describe the build environment. USE flags are a large part in that, but here is a lot more starting from CHOST over toolchain variables (CFLAGS and co, gcc version, ...) up to listing all library versions used, and likely even more. Note that most of this metadata is not used/generated by portage, so there is quite a bit of additional work besides just running `ROOT=/newpackage emerge -b $package` (tbh, it's simply that .tbz2 packages lack a lot of important information).
  The main implication of this is that you get a huge statespace (use flags * cflags * chost * system state), so the reuse rate of existing packages might not be as high as you expect.
  
• dependency mismatches: a general problem when you try to mix source packages and binary packages is that the dependencies don't match in one way or the other. Look at the recent expat problems and project that on a much larger scale.
  

If you are interested in working on this you probably first want to take a closer look at the .tbz2 format and compare it to established binary package formats like rpm or deb (which are IMO generally much better suited for this task). Then try to improve the format and get those improvements into portage. This would make binary package distribution for Gentoo systems in general quite a bit easier.
Of course you can also try to hack on a emerge wrapper to handle all this but in the long term this will become quite a pain I think.

[Matteo Azzali]

Find a safe way to "lock" binary files to not have malware/rootkit inside and this option can become viable.
(MD5 is not an option, in this case, and I fear you'll need to use closed source installation-p2p program
to avoid peoples hacking it easily......). This you solved with fingerprints.

Also, there are some issues regardin use flags: some package can have thousands of combination
(limit case is mplayer, more than 50 use flags: more than 2^50 combinations).
Or you accept to compile everything with all use flags on, or you compile with all flags off,
having them mixed is a big issue,(anybody need to have the same default flags in /etc/make.conf and
nothing in package.use) not telling about the big advantage of gentoo on the other
distros: flexibility , that goes away.... I'm not sure the way you solved is so great: 2^50 combinations
means the foundry is likely to need a lot of cpu power to compile........ and/or users need some way to know
what are the most used flags for that package (just 5 flags is 2^5=32, too much to think to compile all of them).

In the end, another issue: in gento we need to have in the hard disk the compiled packages and
as well we keep in distfiles the source archives. Implementing this binary distribution inside
Gentoo would mean a third layer with packed binaries (if you want to redistribute them after compiling).

So in the end, most of the people would just wait to do "emerge word" after "emerge sync",
to be sure to not use time to compile and space for the packed binaries......
(cause people is lazy!) Some "credits" system would be needed.
_________________
Every day a new distro comes to birth. Every day a distro "eats" another.
If you're born distro, no matter what, start to run.
---- http://www.linuxprinting.org/ ---- http://tuxmobil.org/

[rafo]

Hi Matteo,

I don't follow you, why could not MD5 or similar be used to validate a downloaded binary package?

The installation program must be trustworthy, just like any operating system component that is run with root privileges. But it should be OK if you just use the installation program that the Bonus Binary Foundry provides, and I don't see that it has to be closed source. In principle we have the same situation when we trust the "emerge" program that Gentoo provides.

There are certainly packages that depend on many USE flags. Still, some people will install those programs with Gentoo default flags; they will then have a fair chance that there is already a binary package on the p2p network. Others will have set unusual combinations of USE flags, they will most likely find that the install program resorts to local compilation, no worse than today. Flexibility is just as great as ever.

If disk space is critical you may not want to offer any of it to the p2p space. However, hard disks of today are often over 100 GB in size and offering a fraction of it to the p2p space may not be a big deal at all. At the end of the day, those that don't like the sharing scheme should simply not use it.

I would hesitate to introduce a credit system unless there is really strong evidence that the scheme is abused. One reason is that vintage machines may have fairly small disks, and at the same time benefit greatly from being able to install binaries.

[Matteo Azzali]

[rafo]