Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage with p2p
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
pelux
n00b
n00b


Joined: 21 Jun 2002
Posts: 20

PostPosted: Mon Jun 30, 2003 10:15 am    Post subject: Portage improvement thorugh binary packages p2p sharing. Reply with quote

I have just submitted this enhancement in bugzilla , see:
https://bugs.gentoo.org/show_bug.cgi?id=23747
Here is the text, let me know what you think about it:

----------------------------------------------------------------------------------------------
I know that suggestions similar to this one had alrady been discussed a lot of time, but before you start complaining please read all i have to say.
The portage system is really powerfull. I'm not using Gentoo becouse i think that optimized packages for my system will be a lot faster than precompiled ones (well, maybe :D ), but becouse:
1) the stable branch is really stable but not too outdated (see Debian)
2) i can configure my gentoo installation as i really want (only the packages i want, the way i want)
3) if i have to install packages like mplayer , or the latest loster, i can do this with no trouble at all
The only drawback is the compiling time. Yes, i can compile in the night, but sometimes i really need to have a package installed immedialy, of simply i don't want to wait ages to have kde compiled on my not too powerfull laptop.
Finally here is the suggestion:
The idea is to create a p2p network where gentoo users can transparently share the already compiled packages.
Considering one package "name_version" (let's say kdebase_3.1.2) this can be different from the others already compiled one (samepackage_sameversion) only in three ways:
I) used optimizations
II) used USE flags
III) linked library's versions used (let's say it may be compiled using pam_0.73 or pam_0.74 as the dependency rule only states => pam_0.73)
IV) gcc compiler used
On point I i think that a lot of gentoo users will be happy to renunce using exotic CFLAGS settings, it they can get the packages already compiled THE way thay want with "standard" optimizations like CFLAGS="-march=athlon-xp -O3 -pipe -fomit-frame-pointer". Also the gentoo developers team is moving in the direction of defining some "stable&standard" predefined cflags settings for various cpu, as can be inferred from the "CFLAGS/cpuinfo collection project".
Point II should not be that great problem, as the possible combinations of optional USE flags in packages is not that high number. Also it usally happend that there are some "most used" USE flags combinations.
Point III is probably the major one. We have to distinguish between kdebase_3.1.2 compiled upon pam_0.73 and kdebase_3.1.2 compiled upon pam_0.74 (and also between different releases).
Point IV is not a problem as the gcc version used in gentoo doesn't change frequently.
All this system should be based on an existing p2p sharing protocol (maybe adapted) that has to be efficient, fast, scalable, and working also with firewalled users (there are a lot....) (am i asking too much? :D ).
The key point is the number of users adopting this feature (it would be nice to have an entry make.conf under features , like "binary_sharing"=yes), that should be enaugh large to cover all the needed combinations. The good news is that gentoo's user base is currently growing, with the popluraty of the distro, so i think that using this system a user shouldn't have any problem in findind the packages it need with the right characteristic for popoular (and not only) programs. Kde hell could end! :P
All the sharing have to be completely transparent to the user! So, if a user enables this feature in make.conf , the following two things will happend:
A) for every ebuild compile, two files will be generated under an arbitrary directory:
1) the binary package (ebuild with -k option)
2) the description (.desc maybe?) file, with the same name of the binary package, cointaing the three infos needed (optimizations, USE flags, version/revision of the packages the package depend upon).
This way the user actively contribute to the gentoo sharing comunity in a transparent way [an algorythm that deletes obsoletes file shold be added later......]
B) for every ebuild compile, portage automatically search with a query on the p2p gentoo comunity if the requested package is already present, with the required characteristic. If so it download it and install it.
A md5 check should be added later for security reason.
I know that a lot more should be added and discussed, but i need to know what you think of this idea. I also have to add that i don't have the programming experience needed to be a "first line developer" that directly modify portage and the sharing protocol, but i want to contribute for what i'm able to.
I really think that ,as the user base is already very large, this project would be succesfull.
This enhancement could improve a lot the gentoo experience, given the reduced package's installation time, while preserving all the portage excellent features.
For the brave who dared to read me till this point, i thank them! (also sorry for my bad english....)
Hope that this contribution will help the gentoo comunity.
Bye

Stefano Peluchetti
Back to top
View user's profile Send private message
Carlo
Developer
Developer


Joined: 12 Aug 2002
Posts: 3356

PostPosted: Mon Jun 30, 2003 10:40 am    Post subject: Reply with quote

The answer is simple: NO!

I wouldn't trust another gentoo user just because he uses gentoo.


Carlo
_________________
Please make sure that you have searched for an answer to a question after reading all the relevant docs.
Back to top
View user's profile Send private message
pelux
n00b
n00b


Joined: 21 Jun 2002
Posts: 20

PostPosted: Mon Jun 30, 2003 12:06 pm    Post subject: Reply with quote

But you don't have to trust him becouse you "feel" to do so.
A md5 sum check will be implemented, so you you will be sure that that you are not getting a trojaned version of the program. Anyway the security aspects of this system will be discussed in details if the idea get enough positive feedback. Obviously this point will be taken in grat consideration, as it's clear that you have to be asolutely sure that wahat you are going to install is not compromised.
Thanks for your reply.
Back to top
View user's profile Send private message
ciguanabo
n00b
n00b


Joined: 26 Jun 2003
Posts: 4
Location: The Netherlands

PostPosted: Mon Jun 30, 2003 12:11 pm    Post subject: Reply with quote

I have to agree with Carlo on this. Source code can (in principle) be checked easily, but binaries can't (because of all the different flags, options, etc). A solution could be to have official binaries which could be MD5 checked, of course this would put some restrictions as to how many variants of a binary package are available because not everybody will be able to compile official packages.
This reminds me... what happened to the GRP packages in 1.4rc2? They were really usefull for intalling a desktop system rapidly (and you can always compile the programs later with your own flag and optimalization settings), but I couldn't find them in the latest live-cd versions. Have the packages been deprecated?
Back to top
View user's profile Send private message
patan
n00b
n00b


Joined: 19 Feb 2003
Posts: 66

PostPosted: Mon Jun 30, 2003 1:31 pm    Post subject: Reply with quote

ciguanabo wrote:

This reminds me... what happened to the GRP packages in 1.4rc2? They were really usefull for intalling a desktop system rapidly (and you can always compile the programs later with your own flag and optimalization settings), but I couldn't find them in the latest live-cd versions. Have the packages been deprecated?


This was mentioned awhile ago in the newsletter

I just wanted to take a moment and let everyone know where we stand as far as a release goes. Plans are to release an official RC3 on Thursday the 27th. This rc will include stages from the sparc, x86, and ppc teams. The x86 port will be a limited release meaning a single "one size fits all" compiled set of stages and no GRP will be packaged.

http://www.gentoo.org/news/en/gwn/20030224-newsletter.xml
Back to top
View user's profile Send private message
Pythonhead
Developer
Developer


Joined: 16 Dec 2002
Posts: 1801
Location: Redondo Beach, Republic of Calif.

PostPosted: Mon Jun 30, 2003 5:18 pm    Post subject: Reply with quote

Quote:
A md5 check should be added later for security reason.


Nobody in their right mind is going to use this p2p system until this part is done first. And who is going to keep track of which package is genuine and which is a trojan?

If Gentoo were distributing the binaries for users to share in a p2p network that would be one thing, but your idea is a script kiddies wet dream.

If people are complaining about the official mirrors being slow, wait till they try a p2p system.
Back to top
View user's profile Send private message
pygoscelis
Guru
Guru


Joined: 07 Jun 2003
Posts: 402

PostPosted: Mon Jun 30, 2003 6:05 pm    Post subject: Reply with quote

Here's a rudimentary security system.

We don't share linked binaries, we share object files. Typically there are many object files per shared package (or else it doesn't make sense to share it). Now you select few random files from the bunch and recompile them. If your objects and downloaded objects are identical, consider the entire package good.

This is not 100% bulletproof of course but it's something. Now, bizarrely, TCPA would make it easy to implement a 100% bulletproof check...
Back to top
View user's profile Send private message
pelux
n00b
n00b


Joined: 21 Jun 2002
Posts: 20

PostPosted: Thu Jul 03, 2003 9:32 pm    Post subject: Reply with quote

Sorry for the late reply, but i was very busy with study.
Anyway.....an idea should be to have users register to a gentoo page to be able to use this p2p system.
This way we could have differen type of "accounts",so you can see who contibuted that package, and check if for example it is a gentoo developer (a D flag for developer and U flag for users should be ok).
Also you can have a "trusted circle" of friends (that you know personally) and that you trust, so you can choose to download by them anyway.
Good night

Pelux
Back to top
View user's profile Send private message
dasalvagg
Apprentice
Apprentice


Joined: 26 Jun 2002
Posts: 183
Location: NY

PostPosted: Wed Aug 27, 2003 8:59 pm    Post subject: Portage improvement, use p2p networks Reply with quote

This has probably been proposed before. I dont for a second think this is a new idea....

A vast majority of Gentoo users have broadband internet connections, and a vast majority use portage daily. The nature of Gentoo's source based distribution causes a drain on servers hosting the distfiles. This has been an issue in Gentoo land before, and will continue to be as the Gentoo userbase grows. The bandwidth problem can be eliminated with existing technologies applied to the Gentoo environment. My idea stems from the fact that I, and many other Gentoo users are willing to share a few kb/s, but do not want to dedicate a server to be a mirror. A couple reasons why I don't personally run a mirror.
1. I don't leave my computer on all the time.
2. Bandwidth intensive applications need to utilize full connection.
3. Can't handle a thousand requests a day.
4. Would only like to be able to server 2-3 files at a time.

I, like most other Gentoo users, accumulate a large number of source files in my /usr/portage/distfiles directory. Also, like most Gentoo users my computer has a broadband connection and utilizes very little of its full potential on average. It is a logical choice that ap2p network could alleviate the bandwidth problems. Any potential p2p network must be a voluntary network, to use on either end. By this I mean that if I am going to download a source file, then I must have the option of downloading from a mirror, such as ibiblio, or from the p2p netowrk. And likewise, once I have that file I must have the ability to chose to share the file, or not share the file. The protocol is largely irrelative, as long as it is efficient. Bittorrent, and Gnutella come to mind. In these circumstances it would be to the benifit of Gentoo users to have a seperate p2p network...sidestepping any legal action taken against these.
Key points
1. Use an existing protocol.
2. Voluntary to use the p2p network ie. /etc/init.d/gentooP2P (start | stop | status)
3. Adopt this into portage ie. emerge --p2p gaim Will download off p2p
4. Allow users to stop service if already involved in an intensive operation.
5. By sharing 50kb/s and using something like bittorent we can make a substantial dent on our use of distfile servers.
6. Have backup mirrors, if a file cannot be found on our p2p network, then use one of the main mirrors.

What do people think?
Back to top
View user's profile Send private message
broschi
Apprentice
Apprentice


Joined: 20 Aug 2002
Posts: 189
Location: Atlantide

PostPosted: Wed Aug 27, 2003 9:37 pm    Post subject: Reply with quote

Don't forget that we also need a way to check the authenticity of the file. I guess we might as well simply check the md5sum from the current mirrors and check it against the ones that come from a p2p net. This way we prevent trojan horses in the packages among other things.
_________________
"Is this type of thing going to happen every time we switch to improbability drive?" "Very probably I'm afraid."
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Wed Aug 27, 2003 9:43 pm    Post subject: Reply with quote

broschi wrote:
Don't forget that we also need a way to check the authenticity of the file. I guess we might as well simply check the md5sum from the current mirrors and check it against the ones that come from a p2p net.
Most P2P protocols (Freenet, Bittorrent, and FastTrack come to mind) track files internally by a collection of hash values, so authenticity in that manner isn't a problem.

broschi wrote:
This way we prevent trojan horses in the packages among other things.
Unless, of course, the package is compromised upstream -- BitchX, sendmail (IIRC), and most recently GNU have all had break-ins that resulted in modified files.
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
Qball
Apprentice
Apprentice


Joined: 25 Nov 2002
Posts: 196

PostPosted: Wed Aug 27, 2003 9:47 pm    Post subject: Reply with quote

The idea is good..

I think for us poor adsl upload people it would be nice to have bandwith throttling..
and partial download (download it from 10 host at the same time) would be needed. (because not everybody has a big 10/100mbit upload. )
btw.. as protocoll we could look into gift.
Back to top
View user's profile Send private message
dasalvagg
Apprentice
Apprentice


Joined: 26 Jun 2002
Posts: 183
Location: NY

PostPosted: Wed Aug 27, 2003 10:03 pm    Post subject: Reply with quote

Quote:
I think for us poor adsl upload people it would be nice to have bandwith throttling..
and partial download (download it from 10 host at the same time) would be needed. (because not everybody has a big 10/100mbit upload. )


Yes, these are some of the reasons why I thought existing protocols would be better. As programmers(most of us) we know dont re-invent the wheel. Downloading a file from multiple hosts at the same time is what makes this great, people that have poor upload speeds can still contribute their extra bits.
Back to top
View user's profile Send private message
pytigger
Tux's lil' helper
Tux's lil' helper


Joined: 27 Aug 2003
Posts: 134
Location: Hanover, Germany

PostPosted: Wed Aug 27, 2003 10:31 pm    Post subject: Reply with quote

This idea looks good. But there are drawbacks, other than authenticity:

1. unofficial mirror have to invest traffic / time for keeping up to date, or they are worthless
2. A P2P network is unreliable. I would not want to rely on it, I'd just use the working mirrors
3. Gentoo should make up it's own network. Existing networks, especially younger P2Ps would be severely polluted by the portage namespace!
_________________
And by the way - Gentoo needs an official WIKI!!
Back to top
View user's profile Send private message
syko
n00b
n00b


Joined: 15 Jul 2003
Posts: 36
Location: Alberta, Canada

PostPosted: Wed Aug 27, 2003 10:57 pm    Post subject: Reply with quote

pytigger wrote:
This idea looks good. But there are drawbacks, other than authenticity:

1. unofficial mirror have to invest traffic / time for keeping up to date, or they are worthless


I'm not sure that I understand you correctly, but the point wouldn't be to build an unofficial mirror, we'd, as users of Gentoo and Portage, simply use the system as normal, allowing people to download distfiles from us. For those of us who don't keep constantly up to date with the Portage tree, yes parts of our collection would fall into decay, however, as we add new packages to our collection we would be able to help the network with different distfiles. And if you're the type of person who doesn't add or upgrade any software after you get the system running, then you simply wouldn't need to participate in the network at all.

Plus with the "redundancy" (I know it's a buzzword, but it does describe it well) provided by a p2p network, and the ability to fall back on a standard official mirror, I don't see how reliability could be an issue. However, I do agree that bogging down an existing p2p network wouldn't be very net-friendly of us.

Personally, I think that it is an excellent idea, and using existing protocols (e.g. BitTorrent, among others), on a separated network, this could indeed be an impressive innovation.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9507
Location: beyond the rim

PostPosted: Thu Aug 28, 2003 3:55 am    Post subject: Reply with quote

Someone is working on it:

http://article.gmane.org/gmane.linux.gentoo.devel/10376
Back to top
View user's profile Send private message
dasalvagg
Apprentice
Apprentice


Joined: 26 Jun 2002
Posts: 183
Location: NY

PostPosted: Thu Aug 28, 2003 4:48 am    Post subject: Reply with quote

Quote:
Someone is working on it:

...knew i couldn't be the first

Quote:
1. unofficial mirror have to invest traffic / time for keeping up to date, or they are worthless

WE are the unofficial mirror, but seeing how many of us have a couple hundred packages laying around, then why not put them to good use. Just think, how many packages on your system(that are currently emerged) are over a month old? The majority if you're like most people. This means that those up-to-date packages would be mirrored and available by everyone.

Quote:
2. A P2P network is unreliable. I would not want to rely on it, I'd just use the working mirrors

no one is stopping you....but as bittorent has proven introducing new packages into the system..(ie linux .iso) allows them to quickly circulate while still reducing the network load of the main servers.

Quote:
3. Gentoo should make up it's own network. Existing networks, especially younger P2Ps would be severely polluted by the portage namespace!

Couldn't agree more. We could literally take over bittorent if we all got on at once. Plus it seems that every p2p network is getting sued sooner or later. Dont think we want to be a part of that.
Back to top
View user's profile Send private message
Ari Rahikkala
Guru
Guru


Joined: 02 Oct 2002
Posts: 370
Location: Finland

PostPosted: Thu Aug 28, 2003 5:27 am    Post subject: Reply with quote

dasalvagg wrote:
Quote:
3. Gentoo should make up it's own network. Existing networks, especially younger P2Ps would be severely polluted by the portage namespace!

Couldn't agree more. We could literally take over bittorent if we all got on at once. Plus it seems that every p2p network is getting sued sooner or later. Dont think we want to be a part of that.


Actually, each torrent and tracker is independent. You can't "pollute" the BitTorrent "network" because it does not exist. This is a part of the reason why BT is so efficient... its control is centralized (for each torrent) while the data flow is distributed.
_________________
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Thu Aug 28, 2003 11:54 am    Post subject: Reply with quote

I believe bittorrent is the best option.

Personally i have been thinking of modifying the current bittorrent ncurses cleint to allow me to mirror ISO's more efficiently. Its not a good idea to run a new instance of the client for each .torrent file.

So i was thinking there would be like a small server application. That every couple of minutes checks the tracker to see if there are anyone trying to download the files that the server can serve. Then if there is it will start bittorrent client, to allow that person to download the file from the server.

This is just a basic idea, but i'm thinking of making it so that only one instance of bittorrent is running for multiple torrents.

This would work great on mirror servers.

Anyway, its just an idea. :D:D
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
pytigger
Tux's lil' helper
Tux's lil' helper


Joined: 27 Aug 2003
Posts: 134
Location: Hanover, Germany

PostPosted: Thu Aug 28, 2003 6:47 pm    Post subject: Reply with quote

If you have a dozen or more systems in a network that all use, say, AMD Duron processors, the other components differing more less, and you want to install Gentoo on all of them, if you are not very very clever, a source distribution would take hours, significant updates too.
If the filesharing thingy could be configured to make a local network mirror binary packages, that would be very cool, because every new computer introduced to the lan can install gentoo without the need to compile everything anew. In the internet this might be possible too: Since a P2P network multiplies the gentoo network strength, it can begin to store binary packages for those who want them.
_________________
And by the way - Gentoo needs an official WIKI!!
Back to top
View user's profile Send private message
ebrostig
Bodhisattva
Bodhisattva


Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl

PostPosted: Thu Aug 28, 2003 6:53 pm    Post subject: Reply with quote

Currently no P2P application has a good solution for people behind firewalls, esp restrictive ones like the one I'm behind.

If this was implemented I would have to find another distro to use since I would never be able to update Portage.

Please, Gentoo is not only used in homes and the environment in the corporate world is very different from your home universe.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'
Back to top
View user's profile Send private message
Sieg
Tux's lil' helper
Tux's lil' helper


Joined: 07 Oct 2002
Posts: 87

PostPosted: Thu Aug 28, 2003 7:55 pm    Post subject: Reply with quote

It's a good suggestion, but indeed, the option of being able to turn it off would need to be there. There are a lot of reasons why some people would not like to share their bandwith, and the use of Gentoo could not affect that decision. If people want to do it, it would be a great addition, but the choice needs to remain there.

As ebrostig pointed out, this could not be activated in probably 99% of the corporate installations of Gentoo, as this would not only be bad ethics, but important costs on system usage. Home users would be affected in a couple of ways. First of all, not all ISPs offer unlimited bandwith to broadband users. I for one am stuck with a limit of 15GB of download and 15GB of upload from 7am to midnight. I know some ISPs that have limits even lower than that, with as little as 1GB of upload per month. They could just not sacrifice their small ratio for that. The other way that we could be affected is latency. Maybe there are not that many Gentoo users that are heavy gamers, but anyone that plays over the Internet knows that uploading kills your pings for most connections. Therefore, all these people would not want this service to be available during those times.

My 2 cents.
Back to top
View user's profile Send private message
dasalvagg
Apprentice
Apprentice


Joined: 26 Jun 2002
Posts: 183
Location: NY

PostPosted: Thu Aug 28, 2003 8:54 pm    Post subject: Reply with quote

I've never heard of a 1gig upload limit, but maybe i'm just sheltered. Of course....this was always intended to be an addition to the mirrors that are already in place, and not necessarily a replacement.

Gaming, low upload limits, corporate people should all be able to turn it off with a simply command like
Code:
/etc/init.d/gentooP2P stop
Back to top
View user's profile Send private message
ebrostig
Bodhisattva
Bodhisattva


Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl

PostPosted: Thu Aug 28, 2003 9:18 pm    Post subject: Reply with quote

dasalvagg wrote:
I've never heard of a 1gig upload limit, but maybe i'm just sheltered. Of course....this was always intended to be an addition to the mirrors that are already in place, and not necessarily a replacement.

Gaming, low upload limits, corporate people should all be able to turn it off with a simply command like
Code:
/etc/init.d/gentooP2P stop


Err... no... it's rather have to be like this for everyone who wants to use it:

Code:

/etc/init.d/gentooP2P start


Such a service should not be enabled by default nor forced to be on have someone stop it.

Besides, a P2P is a bigger security risk than anything else you can run on your PC (well, maybe except for any MS product that is :) )

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'
Back to top
View user's profile Send private message
dasalvagg
Apprentice
Apprentice


Joined: 26 Jun 2002
Posts: 183
Location: NY

PostPosted: Thu Aug 28, 2003 9:28 pm    Post subject: Reply with quote

something of interest to this discussion...
http://news.gmane.org/onethread.php?group=gmane.linux.gentoo.devel&root=%3C20030723194823.GJ9959%40mail.lieber.org%3E
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2, 3, 4, 5, 6  Next
Page 1 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum