P2P sharing of locally compiled packages

[rudregues]

Introduction
Gentoo is a source based meta-distribution that has its advantages and disadvantages, and it been for many years around. For installing packages the user need to compile it's source code most of the time, and it has some impacts: compilation takes CPU and RAM resources and can last many minutes or even hours. For many people it's not so significative, but for others it's a dealbreaker. So why not implement some way of using more binaries? Throughout Gentoo history, there has been many discussions about this topic. Some people even tried to implement pre-compiled mirrors, like http://chinstrap.alternating.net/ for example. But this has it's shortcomings: put packages with just common USE flags, use only generic CFLAGS etc. Because put a binary for every combination of CFLAGS and USE flags would be TB and more TB of hard disk server usage, not to mention a possible bandwidth problem. There was some people suggesting a p2p system, that would put less overhead in servers. Maybe a server with just .torrent files? This lead me to the question:

Would it be viable to have a system in which people compile their software and then p2p share it with other?

I searched a lot and found some relevant discussions related to this. In chronological order:
Binary packages for stable users [long]
binary package distribution
Binary package servers (once again ...)
Official Package Repository, WHY NOT?

Disclaimer: I'm not trying to enforce anything, but I think it would be awesome to Gentoo if this works really well. And I think the way I'm suggesting won't change Gentoo's nature like it would if maintaners had a task to keep binaries by themselves (this would approach Gentoo to a binary distro).

The method
The idea here is to have a server with many .config files and it's associated .torrent files for each foo package. To achieve this, I made a sequence of four steps. Note that this is a very naive tackle on the subjects involved, I'm not a developer or specialist in any of them. I even searched how p2p works before posting this. Feel free to correct me if I'm wrong with something.

First step
In the user machine, emerge creates a foo.config for foo package containing a list with:
- CFLAGS (to avoid march and mtune native problems just 'translate' to the correct optimization)
- USE flags
- GCC version

Second step
Emerge generates a hash1 from foo.config and renames it to foo_hash1.config

Third step
Emerge checks the availability of foo_hash1.config in the hash repository:
1) If hash1 exists, emerge will use it's associated torrent file to download foo-bin and then install it
2) If hash1 doesn't exist, then it goes to the next step

Fourth step
1) Package foo is locally builded
2) If the compilation is successful, emerge generates a hash2 for foo-bin and creates an empty foo-bin_hash1_hash2.torrent file
3) Emerge splits off the binary in many chunks and calculate the hashes of each one
4) Emerge fills up foo-bin_hash1_hash2.torrent with the previous hashes
5) Emerge uploads foo-bin_hash1_hash2.torrent and foo_hash1.config to the repository

Some details:
- I'm not sure if in the first step portage version should be checked too
- Emerge could first ask the user if he really wants to upload his binary

Problems with this method
- Another feature in portage means more work for devs, including maintenance, tests, bug fixing etc.
- It would add some GB more of files in the servers and need to create another hierarchy of directories besides the existing one
- This method would require users being able to upload text files to the server somehow (maybe some kind of key generated by emerge at runtime for temporary access?)
- Security concerns: someone could put malware in the binary or the authenticity check could be enough?
_________________
Emerging en gentoo

[Maitreya]

I am currently working on something similar for my personal Gentoo derived distro.
It will be using IPFS for storage and having the TBZ2 packages unpacked in there.
This will deduplicate the files and leave the actual changes to deltas. This also gives other distro maintainers the option to pick up on those files (docs, datafiles etc)
The main reason I am doing this will not be not compile anymore (although it can be a nice sideeffect).
But to prove my binaries are the same everywhere. I.e. prove your binaries are not "infected" (Ken Thompson compiler hack)
And also to easily test/bench differently created binaries.

[ct85711]

The problem with is, is using march=native will break this right away. This is because native looks up what your processor supports instead of doing general groupings, thus 2 computers may not have the same cpu instructions and be an invalid instruction error.

[khayyam]

[rudregues]

[NeddySeagoon]

Maitreya,

Proving the binaris are the same everywhere just shows that all the compilers have the same hack.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Maitreya]

NeddySeagon, correct!

But probably unlikely. Still am interested in the differences I can find.


And khayyam, I've seen it before but thank you for mentioning it as it does fit for a large portion of the work.