Portage with p2p

epall · Posted: Sun May 16, 2004 5:24 pm Post subject:

It seems to me that trying to distribute the portage tree over p2p is fraught with perils. Trying to distribute compiles has even more problems. I think the problem that we should focus on most is distributing source .bz2/.gz files that live in /usr/portage/distfiles.

sanity · n00b Joined: 19 May 2002 Posts: 39 Location: Iowa

the .bz2/etc files in distfiles have the same problems as the portage tree, only less so. What scales best to bittorrent (and not what I just described) is a huge tarball of everything.
_________________
Every time you read a signature, God kills a kitten.

epall · Posted: Sun May 16, 2004 6:12 pm Post subject:

exactly! That's why I don't think we should use bittorrent. A huge tarball would be a massive waste of bandwidth, because people would only need a few megs of it. BitTorrent would be well suited to a huge tarball, but that's not what we want. I think another protocol would be a better choice.

sanity · n00b Joined: 19 May 2002 Posts: 39 Location: Iowa

I wasn't suggesting bittorrent directly. I don't know if there's another protocol that does what... well, what I was suggesting about three posts ago. Scroll up

epall · Posted: Sun May 23, 2004 5:46 pm Post subject:

You're right, but like I said, trying to distribute the portage tree over p2p is not as easy as distfiles. I think that a good system would be using your original idea for just distfiles over a napster network.

mlsfit138 · Guru Joined: 20 Sep 2003 Posts: 406 Location: Washington

Gnunet would probably be more appropriate than freenet. It is written in c++, is searchable, and offers similar data integrity assurance.

Many of you think that it would be a mistake to tap into an existing p2p network, but if gentoo were to use something like gnunet, or freenet, it would lend that network a great deal of credibility, and support those project's goals of promoting free speech.

besides, if this service isn't enabled by default, so what if the network is taken down for legal reasons? (we would still have our mirrors)
_________________
"Everytime you justify
another good in you dies"

-Converge, The Saddest Day, Petitioning the Empty Sky

Squinky86 · Posted: Sat Sep 04, 2004 9:15 pm Post subject:

fava, I've been trying to get in touch with you, but haven't been able to.
For anyone interested, I have some ideas. Please see bug #62331 and happy hacking all

.

21:17 <@squinky86> !seen fava
21:17 <+GenBot> fava was last seen 13 weeks, 1 day, 18 hours, 26 minutes and 34 seconds ago, saying 'seemant: thanks' in #gentoo-dev.
_________________
Me

Lokheed · Posted: Sat Sep 04, 2004 10:03 pm Post subject: Re: Network requirements

Squinky86 · Posted: Sat Sep 04, 2004 10:52 pm Post subject:

Useage of this feature can only be used for gentoo distfiles in the form I am working on- you will be unable to use p2p-portage for anything illegal. It will be a separate package, so if it's illegal to use a p2p program in your country, just don't emerge it.
_________________
Me

nmcsween · Guru Joined: 12 Nov 2003 Posts: 381

why not use http://konspire.sourceforge.net/ as a backend and modify it a bit to work with portage?
_________________
Great Resources

Fluid Portage

Squinky86 · Posted: Sun Sep 05, 2004 12:32 am Post subject:

trexmaster · n00b Joined: 13 Sep 2003 Posts: 17 Location: Paris, France

[quote="Lokheed"]

gentoo_lan · Posted: Wed Nov 24, 2004 10:35 am Post subject:

SPY_jmr1 · n00b Joined: 02 Jan 2004 Posts: 62

but, isn't the GPL a copyright?... and if... we can't share copyrighted things, ... then... we can't... but its... but they... but. . .

*KERBOOM!*

Squinky86 · Posted: Thu Nov 25, 2004 12:11 am Post subject:

No files are being mirrored on the gentoo mirrors that would violate an unredistributable copyright. The same will apply to p2p distfile mirrorage. And it will be OPTIONAL: the way I see implementing it is by emerging the right package and setting the FETCH_COMMAND in /etc/make.conf.
_________________
Me

dasalvagg · Apprentice Joined: 26 Jun 2002 Posts: 183 Location: NY

i'm again interested in this area as I have newly found free time. Is anyone working on this currently? Is there proof of concept code?

RedDawn · Posted: Thu Feb 03, 2005 10:10 pm Post subject:

Any updates i would be happy to test this out.

davidsb · Posted: Tue Feb 15, 2005 2:47 am Post subject:

I don't have the skills nor the time to put a good effort on this project, but I've been following the interesting discussion and I'm for sure interested on testing or whatever else comes up that I migth help out.
_________________
http://recycler.homelinux.org/~wolfshade/bootlegs/

travail101 · Posted: Sat Apr 02, 2005 10:26 pm Post subject: swarming in any shape or form

after throwing the apparently old tired p2p idea into #gentoo on Freenode, seeing interested users, and then promptly being directed to the forums to look at many failed attempts at finding a worthy solution... I thought of a few more ideas.

My ideas may be a bit presumptuos, or forward, or half-baked, but try and here the concept more than the implementation as I lay it out

DISCLAIMER: sorry i didn't read every post in every thread on sharing bandwidth be it BT or some other implementation of p2ping. secondly this idea is being posted here first instead of in a new thread to avoid being flamed for saying the same thing over and over... if it's new enough I may make a new thread for it... once again with the not reading everything in the entire Forum to know if this is new or not. thirdly... this is not an idea to add existing technologies directly to Portage/Gentoo it's an idea to expand existing or create new technologies that would benefit the entire server world, and would then once created be implemented into gentoo if enough people can be convinced.

ok so now that that's out of the way here's the idea. (the word SwarmHash will be used as the concept of a file pointing to a hash and a tracker, similar to torrent files for BT)

the protocol can be new or adapted from existing technologies, it would need at least three things, a stand alone implementation, an FTP extension, and an HTTP extension... (perhaps plugins for existing FTP/HTTP server software like Apache) the last two needs are so that userless "SwarmHash" files can see FTP and HTTP mirrors as other users, this has more than one advantage, some of which I may not even have thought of, but a few I can see are A) no need to fallback on FTP or HTTP if whatever P2P protocol used in Portage has no users... instead the Gentoo Mirrors would be seen as seeds or other users. B) FTP and HTTP mirrors can all keep each other up-to-date via the same swarm, this is a protocol that if implemented could reach Epic proportions, with massive global swarming... and C) well there is no C yet... you tell me if you see any more advantages to this thought. the non HTTP/FTP implementation is of course just the p2p protocol itself and should exist standalone for the sake of keeping FTP/HTTP server plugin sizes down, and for use by people not trying to use it on a grand scale... it may be modified BT, or if may be the next generation of BT and replace it alltogether. the client software needs to be able to run 24/7 and needs to be able to be set to only use idle time, it needs bandwidth throttling, it certainly needs resume, it needs MD5 or other security Hashing built-in, and so on... if possible it should have a dynamice SwarmHash file that can change what files are part of the download.

the FTP/HTTP server integration is key, and is necessary... if this part is NOT possible the whole idea is almost useless...

how Gentoo would fit in if said software and implementation existed?
yes, you can put something like P2P_ENABLE="yes" in make.conf. this would then use the new protocol for downloading files...

the SwarmHash file should be able to be created dynamically with information from portage, even the portage DataBase, a emerge p2psync could exist if needed to update your Portage DB with the info needed by the new p2p client to make it's own SwarmHash file. there's a few ways this could go, and this is probably going to be the biggest issue to decide on if such an solution could be fabricated, how many files the SwarmHash file should manage: my idea is it should manage a complete Portage database, this is why it must be dynamic, if it's one file that lists all files available in portage then it not only needs to be a dynamic implementation but should be created to take advantage of existing databases, like the one portage uses. this i think is the best way to go, nobody wants 400 processes sharing 400 files from there /usr/portage/distfiles dir, one big file that hardly anyone would ever have a 100% of except servers would be ideal if it can be made to take advantage of external databases, like the users portage database. from the clients view it would look like a ton of users with 5-10% of the total SwarmHash file... the client would accept commands to only get a specific file from a simple command, emerge would create this command dynamically, when you type emerge foo with P2P_ENABLE="yes" in mak.conf emerge will send a command like "swarmhash_get -f foo-0.6.5.tar.gz --SwarmHash=portage_distfiles.s2h" this would tell the client to get the file foo-0.6.5.tar.gz from the SwarmHash file portage_distfiles.s2h (which is our Torrent type file with the information on all file in portage including the one we want to merge right now) the SwarmHash file could even be made to check files and version numbers from the server instead of the local database so emerge could simply send the command "swarmhash_get -f foo -v x86 --SwarmHash=portage_distfiles.s2h" to the client which would ask the trackers server what the latest available version is that matches the query, the -v command would be like --variable which could be compared to a server side list of variables, in this case it's the Gentoo ARCH types... so emerge could send ~x86 for the latest unstable version in portage. if this second scenario were implemented, then there would no longer be a need to resync your local portage database to get the latest version... this second implementation may require a lot more work than I think at this point, either way, the first implementation is just fine too.

so that's all I have right now... p2p with HTTP/FTP extensions so mirrors can swarm and users can swarm, cuts the need for volunteer seeders as the current mirrors would simply need to be convinced to use the new protocol extension for their server software... this should be desireable as the more servers using it, the less time it takes to keep themselves updated, certainly a lot of thought needs to go into this, but if it hasn't already been proposed elsewhere, I think it's a good idea... once again, if FTP and HTTP extensions are not possible for some reason, this whole idea is gonna take a big hit, but it would still be possible... just harder to get servers onboard to seed... anyway, what do you people think?

this is brainstorming time btw, not flaming time, constructive critisism only please. and once more... try to see past my implementation scenarios to the concept of the software as a whole before saying it won't work.

walrus_55 · Posted: Sat Apr 02, 2005 11:42 pm Post subject:

What are these bandwith problems you talk about? i download from portage perfect at 160-170 kB/s every time i can never get that on any p2p software even bittorrent the highest ive ever

got is 80kb/s. I think as long as we have servers who are happy to be mirrors then its a much better option. The only thing that could be handy as far as i can see is the ability to stop a download

and resume it later, but hey my longest download is 20 mins anyway.

LucaSpiller · Posted: Tue Apr 05, 2005 6:36 pm Post subject:

There aren't any major problems at the moment, but if Gentoo continues to grow then the demand on the servers will grow and there will be problems. Bandwidth isn't necessarily the problem, rsync causes a fairly high load on the server. Also the main problem with bandwidth is that it isn't free, somebody has to pay for it so wouldn't you rather put a hand in to lower their costs?

OT: The reason why you are probably getting slow speeds with BT is because you are a) downloading unpopular torrents or b) haven't set up port forwarding correctly. I have a 120kb/s connection and get that nearly everytime.
_________________
:: Luca :: Mac Fag :: Original Macbook, 2g RAM :: Closet Linux user (seasoned with salt and pepper) :: C2D E4400 @ 2ghz, 4g RAM (only 3.2g detected under 64bit...), Nvidia 9600GSO ::

varpath · n00b Joined: 02 Feb 2005 Posts: 18

Hi all,

I tried looking around, but it would appear that no one has suggested this idea before.

Instead of grabbing distfiles and emerge --update world from a central server, why cant we decentralise this task to all gentoo users.

I have an unlimited 256k dsl account, I definitely dont mind using my bandwidth to upload my distfiles to other gentoo users, especially when I'm away from the computer. (Another idea, how about using my idle CPU time to help other gentoo users reduce their compile time?

)

I suppose that this would be a good way to show the world that p2p is not just good at sharing porn, warez and (for riaa) illegitimate mp3 and movies. (I'm grabbing smashing pumpkins mp3s from http://rspaa.niluje.net/data , btw)

Just a though.

d4rkwingduck · Posted: Sun Aug 28, 2005 9:05 am Post subject:

Shadow Skill · Veteran Joined: 04 Dec 2004 Posts: 1023

None of these are bad ideas but security issues aside one has to wonder if concepts like folding@home could really be applied to compiling software since these are not jus mathematical computations but build operations that have various features enabled for a given cpu. It would be nice if it could be done however.
_________________
Ware wa mutekinari.
Wa ga kage waza ni kanau mono nashi.
Wa ga ichigeki wa mutekinari.

"First there was nothing, so the lord gave us light. There was still nothing, but at least you could see it."

aidanjt · Posted: Sun Aug 28, 2005 9:57 am Post subject:

Thats a pretty good idea actually, and we don't really need to worry about 'security', since we fetch the md5 file for packages from the offical rsync mirror. bittorrent transfers are pretty efficent when everyone chips in, think of how fast grabbing Gentoo LiveCD's with bittorrent is.. its a pretty good system, and everyone win wins all around.

not sure about internet distributed compilations thou :/