Selective emerge rsync to cut down on bandwidth usage

OdinsDream · Veteran Joined: 01 Jun 2002 Posts: 1057

When you execute emerge rsync, files are downloaded for every single ebuild in portage.

I myself, however, don't use nearly a quarter of these files, and some of them I'll never use (like the KDE stuff...), but I still have to download these files.

Is it necessary for portage to operate this way? Once the scale of portage continues to grow, the time required for an emerge rsync will increase significantly.

For instance, I emerg'ed samba, but halfway through the compile, noticed the GLSA announcement. I cancelled the compile, and then had to wait another 10 minutes for rsync to update everything, just to grab the latest Samba.

Why not have the ability to say emerge --latestVersion samba, then emerge checks with an rsync server, downloads the latest samba ebuild, and continues from there?

Dependencies for the selected package would also be checked against the server, and if outdated, their ebuilds would be downloaded as well.

masseya

mooman · Posted: Fri Nov 22, 2002 5:59 am Post subject:

Or at least let us rsync up "branches" at a time. Like I'm working on getting my spam proxy box up and running... for me, something like emerge rsync net-mail would be nice, since there are several packages in there that I would like the latest of. Then, since I never run gnome or kde, I could skip ever rsyncing their directories again....
_________________
Linux user off and on since circa 1995

masseya · Posted: Fri Nov 22, 2002 6:01 am Post subject:

mooman · Posted: Fri Nov 22, 2002 7:17 am Post subject:

Tristam,
You saying that 'rsync.mask' is something that works now? or we're just coming up with some compelling reasons for it to be added in?
_________________
Linux user off and on since circa 1995

masseya · Posted: Fri Nov 22, 2002 7:22 am Post subject:

jondkent · Apprentice Joined: 26 Jul 2002 Posts: 289 Location: London

Hiya,

this all sounds like a good idea. Another possibility is to not keep any of these files locally at all, just have a db or flat file that is sync with the apps/versions/etc that are available listed. Then if you wanted to install something you would then sync down the ebuild script and go on from there. That would make life easier for people over 56k lines and remove massive amounts of traffic from the rsync servers (thereby lowering Gentoo costs). After all, as has been said, most people don't use any way near half of the build that are available.

The reason why I suggest this is because people can be lazy (I know I can be) and won't bother filtering down their rsync, esp if they have DSL. The method ensure you only download what is applicable to you

Comments?

Jon

OdinsDream · Veteran Joined: 01 Jun 2002 Posts: 1057

masseya · Posted: Fri Nov 22, 2002 6:54 pm Post subject:

OdinsDream · Veteran Joined: 01 Jun 2002 Posts: 1057

I admit I don't know much about how dependencies are calculated. The suggestion I had would be...in this flat file, you'd have, say,

kde-base
--kde-arts
--kde-libs

gnome-base
--gnome-libs

Or something similar to this, where dependencies are listed as part of the package entry. This is, of course, absolutely necessary. I always pass --pretend to emerge before I actually install a program. To lose that functionality just wouldn't be acceptable. That's what would happen, it seems, if the ebuild were first downloaded, and then dependencies were calculated.

jondkent · Apprentice Joined: 26 Jul 2002 Posts: 289 Location: London

KiTaSuMbA · Guru Joined: 28 Jun 2002 Posts: 430 Location: Naples Italy

My 2 cents on the matter:
I'm totally against the db thing for a series of reasons:
- adding more complexity to the system, double entries for deps (both the db and the ebuild, or perhaps taking the deps from the ebuild and putting on the db which would then make it a pain for rapid or "DIY" ebuild write-ups)
- losing transparency: right now I can just cd to /usr/portage and have a feeling of what is built, what could be built and what is fresh out of the oven... Even more importantly I sometimes modify ebuilds before emerging to change some configure switches that are not taken into consideration using the USE flags. NowIf I had to do that with a db... I really don't know how... You are in definite need of some front end which you can't be sure it will cover everybody's needs (I'd bet it wont actually).
- a flat file is the way to slugginess... so a real db will be necessary (berkleydb terrors??? Don't even say SQL just to use portage...)

What I'm all for is the "branch rsync" idea. Either with a .mask file or at command line option (like emerge rsync net-misc). Unfortunatelly some updated packages will require updated others in another branch that, since they are not rsynced, the emerge will fail to find and satisfy. A solution to that would be, that if emerge finds no adequate version in the tree for a dependency, it forces an rsync in the dep's branch as well before moving on. So it's a bit more work behind the curtains, but it keeps things pretty much as they are while adding a nice feature.

So, what do you think?
_________________
Need to flame people LIVE on IRC? Join #gentoo-otw on freenode!

puddpunk · l33t Joined: 20 Jul 2002 Posts: 681 Location: New Zealand

I thought rsync worked by:

Downloading a file list of the dir to be sync'd from the remote server
comparing that file list to the local dir
deleting/downloading the differences

If it doesnt work like that...why not?

psharp · Posted: Sat Nov 23, 2002 9:14 am Post subject:

dreamer3 · Guru Joined: 24 Sep 2002 Posts: 553

I have a 56k winmodem (yuck) and doing a emerge rsync takes less than 2 minutes with a few updates out there, maybe 3-5 with a LOT of updates (read: I haven't synced up in a while).

Where is the problem? You guys on high speed links should have it WAY better than this.

What gets to me is the apparent complexity increases in portage with each new release. I like the new features and I know we're accomplishing good stuff but every time I upgrade it seems that searching for an ebuild or calucating dependencies takes longer than before, and I know I'm not just imagining things.

scocou · Posted: Sun Jan 12, 2003 6:30 am Post subject:

I think it would be interesting if a user could decide which branches of the portage tree were sync'd with the USE variables. It's a system already in place that abstractly describes what 'type' of err... package support the user needs/wants, and it wouldn't require the user to make reduntant selections in two configs (not to whine it would just be more convenient). Sorry to be so non-technical, it's just a casual thought....

scocou · Posted: Sun Jan 12, 2003 6:31 am Post subject:

I think it would be interesting if a user could decide which branches of the portage tree were sync'd with the USE variables. It's a system already in place that abstractly describes what 'type' of err... package support the user needs/wants, and it wouldn't require the user to make reduntant selections in two configs (not to whine it would just be more convenient). Sorry to be so non-technical, it's just a casual thought....

scocou · Posted: Sun Jan 12, 2003 6:31 am Post subject:

hmm.. said message failed to send but duplicated it 3 times instead! No delete button for some reason, aargh! Sorry all, not my fault...

ahurst · Posted: Tue Feb 13, 2007 3:55 pm Post subject:

Hello,

I share the original poster's idea, and wrote about it elsewhere:
https://forums.gentoo.org/viewtopic-p-3904172.html#3904172

Portage can really be optimised by cutting out the stuff it doesn't need to consider.

If we only locally store the ebuilds for installed packages, but also locally store a full list of valid package atoms in the server's tree then:

- 99% of the tree needn't be stored locally; emerge --sync need only consider installed packages and re-read the tree's package atom list.

- calling 'emerge -p' on a new package reads its portage subtree from rsync server into memory (tiny bw), and can calculate all dependencies recursively thus

- doing an 'emerge <newpackage>' downloads the package's subtree into the local tree, and thus recursively the same for its dependencies

- 'emerge -C' takes away the package's local subtree

Much lower bandwidth, and storage requirements all round.
Can someone defeat this?

Andy

herka · n00b Joined: 08 Aug 2003 Posts: 23

What about this http://gentoo-wiki.com/TIP_Exclude_categories_from_emerge_sync?
Herka

ahurst · Posted: Tue Feb 13, 2007 6:56 pm Post subject:

Yes, quite,

the method in question is hard work. Why not just grep -v your installed packages from the tree (complete package atom list). It does cut bandwidth and processing requirements though.

the second (linked) tip, to store the portage tree in a squashfs is really overkill, because you're still storing 99% of the tree you don't need.

Combine the two and you've got it.

But my idea would kill the two birds with one stone. The trick would be to only ever rsync a subset of the tree, as defined by your request (sync <installed package list>, or emerge <new package>).

I don't know whether rsync can do this. Perhaps I should find out!

Andy