BitTorrent and Portage

Message

Pythagoras1 · Post by **Pythagoras1** » Tue Aug 17, 2004 4:24 pm

what do you think about implementing 'bittorrent' support into portage as alternative to downlading files from mirrors using 'wget'?

i think it would relieve the gentoo mirrors and might also be a speed advantage for broadband users. the torrents could also be shipped inside ebuilds.

kamagurka · Post by **kamagurka** » Tue Aug 17, 2004 4:32 pm

i'll second that in a jiffy. might be a bit hard to implement, but will definetely be good for the mirrors (especially with big packages).

kamagurka · Post by **kamagurka** » Fri Aug 20, 2004 4:04 pm

nobody interested? like, at all? do you people take the mirrors so much for granted?

ashtophet · Post by **ashtophet** » Fri Aug 20, 2004 4:09 pm

im a bittorrent lover, so... it would be really good!

nevynxxx · Post by **nevynxxx** » Fri Aug 20, 2004 4:32 pm

I would have thought that getting individual projects to bittorrentify their downloads, then adding support for that as an alternative was a better way to go about it. It really is not up to us to decide how the source get's distributed. Remember there is non-freely distributable stuff in there.

Bittorrenting the portage tree would be a nice idea. Pretty damned impossible at a guess as it changes so much, and you never want to download it all, just the changes, but a nice idea.

Pythagoras1 · Post by **Pythagoras1** » Fri Oct 08, 2004 12:45 pm

nevynxxx wrote:I would have thought that getting individual projects to bittorrentify their downloads, then adding support for that as an alternative was a better way to go about it. It really is not up to us to decide how the source get's distributed. Remember there is non-freely distributable stuff in there.

i see it the same way but bittorrent and regular downloads don't exclude mutually. the gentoo mirrors could set up bittorrent nodes to.

Bittorrenting the portage tree would be a nice idea. Pretty damned impossible at a guess as it changes so much, and you never want to download it all, just the changes, but a nice idea.

i don't think the distribution of the portage tree uses so much bandwith that bittorrent would be worth a thought for it. the huge bandwith amout comes from the distfiles i'd say. in theory everyone could be able to share his distfiles over the bittorrent network (if wanted).

FishB8 · Post by **FishB8** » Tue Oct 12, 2004 2:44 pm

Ah... Found some people who are thinking the same way I am.

One would assume that using bittorrent could reduce much of the load on the distfiles servers by shifting the load off for the more popular packages.

How hard would this be to implement? Portage would have to be modified, but would bittorrent need any modifications?

Would it be possible to set a USE flag that allows for bittorrent to be installed as part of portage when emerging portage?

nevynxxx · Post by **nevynxxx** » Tue Oct 12, 2004 2:52 pm

FishB8 wrote:Ah... Found some people who are thinking the same way I am.

FishB8 wrote: How hard would this be to implement? Portage would have to be modified, but would bittorrent need any modifications?

I would guess not too hard at all, there are already options in /etc/make.conf for using things other than wget to download files, we would simply be adding another of these.

The question would be more, where does portage find the .torrent files? (i think that is correct, I'm not too up on bittorrent). Possibly an addition to portage that allows the link to the torrent to be included in the ebuild.

FishB8 wrote: Would it be possible to set a USE flag that allows for bittorrent to be installed as part of portage when emerging portage?

Easy, it means adding the word "bittorrent" to a single line in a single ebuild file. Once you have the portage system set up to be able to use bittorrent the use flag is a triviality.

FishB8 · Post by **FishB8** » Tue Oct 12, 2004 4:19 pm

The question would be more, where does portage find the .torrent files? (i think that is correct, I'm not too up on bittorrent). Possibly an addition to portage that allows the link to the torrent to be included in the ebuild.

I'm not "too up on bittorrent" either. From what I understand though, all you would have to do is tell bittorrent to get file x.tar.bz and it goes and finds it for you. It might be more simple that usng wget or ftp because you don't have to worry about addresses. The only problem I could think of is bittorrent fetching a file that matches the name but is still the incorrect file. It's not a problem after downloading because portage checks with an MD5 checksum, but if bittorrent kept delivering the wrong file, you would have to have portage fall back on the standard way of fetching files.

Pythagoras1 · Post by **Pythagoras1** » Tue Oct 12, 2004 9:49 pm

nevynxxx wrote:The question would be more, where does portage find the .torrent files? (i think that is correct, I'm not too up on bittorrent). Possibly an addition to portage that allows the link to the torrent to be included in the ebuild.

why not putting the torrent file into portage? at least this makes sense for large packages but might be overkill for smaller files (<1 MB).

FishB8 wrote:I'm not "too up on bittorrent" either. From what I understand though, all you would have to do is tell bittorrent to get file x.tar.bz and it goes and finds it for you. It might be more simple that usng wget or ftp because you don't have to worry about addresses. The only problem I could think of is bittorrent fetching a file that matches the name but is still the incorrect file. It's not a problem after downloading because portage checks with an MD5 checksum, but if bittorrent kept delivering the wrong file, you would have to have portage fall back on the standard way of fetching files.

the torrent file makes shure it looks for the right file. it checks the checksum *before* downloading.

FishB8 · Post by **FishB8** » Wed Oct 13, 2004 1:20 am

the torrent file makes shure it looks for the right file. it checks the checksum *before* downloading.

Oh I figured BitTorrent has some sort of checking for files, but that's not the problem I was eluding to. Imagine if you ask BitTorrent for a file, and it finds it. It finds the file out on the net by the same name in several locations. It looks at the checksums of those packages, all of them turn out to be "x". Great, they are all the same files, so it proceeds to pull those files all at once from the various locations. Then it hands off the package to portage. However, lets say that when the ebuild was made for the package, the tar ball that it was made in reference to was the same except for something trivial. Say for instance a file was exactly the same but had a different creation date on it, resulting in a different checksum. So now the ebuild file is expecting a tarball with a checksum "y". Oops. Checksum mismatch even though the contents of the tarballs are for all purposes exactly the same; portage refuses to build the file because of differing checksums.

We will probably need a way for portage to tell BitTorrent "I need this file, and it must have this checksum". This is why I was wondering if BitTorrent would have to be modified as well. For one, I don't think there's any way to give it a specific checksum to use. (I may be wrong, I've only briefly looked at it.) And even if there is, we have the problem that portage uses MD5 checksums and BitTorrent uses SH1 checksums, IIRC.

Post by **Pythonhead** » Wed Oct 13, 2004 1:36 am

Pythagoras1 wrote:what do you think about implementing 'bittorrent' support into portage as alternative to downlading files from mirrors using 'wget'?

Everyone would try it and go back to rsync because the number of seeders would be too small and the download rate would stink is my guess.

That and the fact that this has been suggested many times and nobody has written any code to do it doesn't make me think I'll get to test my theory any time soon.

i think it would relieve the gentoo mirrors and might also be a speed advantage for broadband users. the torrents could also be shipped inside ebuilds.

Heres something we can all do to relieve the gentoo mirrors starting right now, today:

Only emerge sync when you need to. Take that emerge sync out of your crontab.

FishB8 · Post by **FishB8** » Wed Oct 13, 2004 1:59 am

Everyone would try it and go back to rsync because the number of seeders would be too small and the download rate would stink is my guess.

Not if the http/ftp servers for the distfiles were also turned into bittorrent sources as well. At worst the rate would probably be pretty much the same. The positive side to this would be if you could get enough momentum built to get it started, more people would probably be interested in participating and offering the contents of their distfile directory on bittorrent.

That and the fact that this has been suggested many times and nobody has written any code to do it

So I've noticed. We need to find some people good at python (i.e. not me) with an interest in both bittorrent and portage.

I guess the logic for arguing in favor of trying something like this is to reason that offloading bandwidth will free up resources for the Gentoo community allowing them to be used elsewhere.

Pythagoras1 · Post by **Pythagoras1** » Wed Oct 13, 2004 8:15 am

Pythonhead wrote: Everyone would try it and go back to rsync because the number of seeders would be too small and the download rate would stink is my guess.

no, i meant only the distfiles not the portage tree.

nevynxxx · Post by **nevynxxx** » Wed Oct 13, 2004 9:06 am

Pythagoras1 wrote: why not putting the torrent file into portage? at least this makes sense for large packages but might be overkill for smaller files (<1 MB).

The threads about "why does emerge sync take so long?" and "why so many files in portage" even if you decide only 10% of cases are deserving of torrent files, thats another 9000 files going into portage at this moment in time.

[OT]

Pythonhead wrote: Heres something we can all do to relieve the gentoo mirrors starting right now, today:

Only emerge sync when you need to. Take that emerge sync out of your crontab.

We are talking about the download mirrors, not the sync mirrors. In most cases they are different.

And considering the sync mirror says please don't use "more than once or twice a day" my daily download, should be quite ok.

As I've said before, if the people who own the mirror ask me to stop syncing so regularly, I will.

Post by **Pythonhead** » Wed Oct 13, 2004 3:49 pm

Pythagoras1 wrote:
Pythonhead wrote: Everyone would try it and go back to rsync because the number of seeders would be too small and the download rate would stink is my guess.
no, i meant only the distfiles not the portage tree.

You're right, I meant to say they'd go back to the normal mirrors.

Post by **Pythonhead** » Wed Oct 13, 2004 4:04 pm

Running an http or ftp server to mirror that rsyncs against a main mirror is one thing, but running trackers for all the files in distfiles isn't the same.

Since the mirror gets updated every two hours or whatever, how are these mirrors going to handle that? Kill the existing tracker and restart it with all the changes to the mirror?

How many people would actually turn on this feature? You do an update world and download 10 files. You start a bit torrent process for each one of these 10 files. How many people are going to keep these 10 programs running after the downloads finish?

piffle · Post by **piffle** » Wed Oct 13, 2004 4:15 pm

Yeah. No thanks.

teknomage1 · Post by **teknomage1** » Wed Oct 13, 2004 4:28 pm

bittorent also brings up open port and firewall issues especially for college students behind university firewalls, and business or corporate users who must abide by established policies.
On a side note, though it is faddish to emerge sync every day, what is the benefit? You still have to wait for the package to compile and gentoo-portage.com provides a convenient way to find the latest version of packag. Why not just sync before you install something?

FishB8 · Post by **FishB8** » Thu Oct 14, 2004 3:46 am

Running an http or ftp server to mirror that rsyncs against a main mirror is one thing, but running trackers for all the files in distfiles isn't the same.

Since the mirror gets updated every two hours or whatever, how are these mirrors going to handle that? Kill the existing tracker and restart it with all the changes to the mirror?

This shows how little I understand BitTorrent. First, I figure that the mirrors wouldn't have much of a problem if the master site that all the mirrors work from took care of the .torrent file creation. Second, do you really have to kill the tracker for it to update changes? That sux.

How many people would actually turn on this feature? You do an update world and download 10 files. You start a bit torrent process for each one of these 10 files. How many people are going to keep these 10 programs running after the downloads finish?

I assume if they were set up to run as a daemon in the background it wouldn't be too much of a problem. You would need a tracker and a stripped down minimalistic web server. (I think that's all isn't it?) The people who don't want to enable such daemons are probably the same people who wouldn't use it anyway.

However, you make a good point. It's sort of a novelty idea. People wouldn't really have incentive to use such a feature over the current setup other than out of personal interest.

bittorent also brings up open port and firewall issues especially for college students behind university firewalls, and business or corporate users who must abide by established policies.

Another good point.

Post by **Pythonhead** » Thu Oct 14, 2004 4:22 am

FishB8 wrote: First, I figure that the mirrors wouldn't have much of a problem if the master site that all the mirrors work from took care of the .torrent file creation. Second, do you really have to kill the tracker for it to update changes? That sux.

You can startup a new copy of the program as files are added to distfiles, but I imagine that would be pretty crazy if you have a lot of files. I think the way most people who seed many files use btlaunchmanycurses.py to do them all in one shot.

I assume if they were set up to run as a daemon in the background it wouldn't be too much of a problem. You would need a tracker and a stripped down minimalistic web server. (I think that's all isn't it?) The people who don't want to enable such daemons are probably the same people who wouldn't use it anyway.

If you're talking about users, they wouldn't need all that. They'd use btdownloadcurses.py instead of wget, but they'd have one running for each program they download. Normally portage waits until wget exits after a download is complete, but with btdownloadcurses.py portage would have to be modified to watch for the file to be done downloading because the torrent client won't exit.

Also, if you're downloading a torrent and its intterupted, you can't resume with ftp or http because torrent downloads don't save from byte 0 till the end. You'd have to remove it and start from the beginning. And there is no timeout for a torrent, unlike http or ftp, so thats another modification to portage.

Theres lots of little things that would have to be worked out.

nevynxxx · Post by **nevynxxx** » Thu Oct 14, 2004 7:57 am

teknomage1 wrote:On a side note, though it is faddish to emerge sync every day, what is the benefit? You still have to wait for the package to compile and gentoo-portage.com provides a convenient way to find the latest version of packag. Why not just sync before you install something?

Well put it this way. I have a scrypt that syncs every night ( it also runs GLSA-Check -p new, and syncs all the BMG rsyncs ) on this machine the portage sync alone takes 15-20 minutes (part of the script times it for me, I get all the results in an email).

When I get into work, I run emerge -uDav world, usually 5-10 packages want to be updated, usually including any GLSA's. Thats why I sync every night.

But hey, thats just the way I work.

coelho · Post by **coelho** » Thu Oct 14, 2004 8:13 am

Pythagoras1 wrote:what do you think about implementing 'bittorrent' support into portage as alternative to downlading files from mirrors using 'wget'?

i think it would relieve the gentoo mirrors and might also be a speed advantage for broadband users. the torrents could also be shipped inside ebuilds.

want to do something usefull ? write some ebuilds and help noobs like me on the forums. Not very good ideia

Post by **kallamej** » Thu Oct 14, 2004 8:41 am

Moved from Portage & Programming as it is not a support request.

ewan.paton · Post by **ewan.paton** » Thu Oct 14, 2004 11:50 am

i dont know about for packages but downloading a bt for rsync could be resonably easy to impliment as torrent clients scan existing files before downloading, what sort of bandwith an emerge sync uses is unknown to me but it may be more efficent but i would guess it would also be slower

BitTorrent and Portage

BitTorrent and Portage

Re: BitTorrent and Portage

Re: BitTorrent and Portage

Re: BitTorrent and Portage

Re: BitTorrent and Portage