emerge sync is too long

[ciaranm]

[t011]

Ok, I'm not sure where our communication problems arise from. But I'm beginning to believe that it's simply because my idea of what portage could be is so different from its current incarnation. Before I get to the two examples you illustrated, let me take a step back and try to explain this in a different way. In many ways what I'm proposing is analagous to the changes that are going on within HTML. As web pages have become more sophisticated developers began adding scripting, css and a host of other elements to the page. That produced a page that served many purposes but was difficult to maintain and to read. Then comes along the idea to separate out everything into different files in a modular approach (XML,XSLT,etc.). That is what I'm suggesting here.

From a practical standpoint it appears that the point you are trying to make from your two examples is that dependencies are difficult to deal with and many different flavors of a package can be made. If that is your point, I agree with you. But it's my opinion that this is a shortcoming of portage, it doesn't have the ability to resolve complicated dependencies. Rather than each package maintainer having to cobble together their own scripting mechanism to deal with the various dependency and flavor headaches, that should be implemented within portage. Let the ebuilds serve as merely the list of dependencies and options that, in a sense, get parsed by portage. Then portage calls individual scripts that are downloaded like distfiles when the package is built for the installation pre and post processing.

Right now ebuilds basically serve 3 purposes.
1) They store all the dependency and flavor info
2) They contain the logic to sort out USE flag conflicts
3) They contain a lot of pre and post installation scripting

My suggestion is to leave #1 in the ebuilds. Move the logic for #2 into portage in a standardized way. And move #3 into subordinate scripts that are downloaded at build time. In this way ebuilds will only do one task, outlining dependencies.

I won't waste any more of your or my time trying to explain this. It's possible that we're simply on such different pages that I can't communicate this to you.

[ciaranm]

[Chaosite]

This is a moot point you're arguing.

Its been agreed already that the major problem with 'emerge sync's is not downloading ebuilds, its generating the cache.

So making ebuilds smaller won't help much, since you still need to cache all the dependencies. Using a database to store the cache is a good idea, even though reiser4 for /usr/portage still works better.

BTW, the problem also when trying to calculate "emerge world -uDav" with a decently large and complex worldfile...

[itsr0y]

Thank you, Chaosite. This is the kind of explanation I am looking for. A simple, straightforward reason is all I want and now I have.

Thanks,
itsr0y

[star.dancer]

So what does the updating of the cache do? I tried to figure it out from the portage code but it's not very clear what's happening to me.

Here's my understanding of it:
- We emerge sync, getting all the ebuilds.
- We use emerge *pkg*, emerge -u *pkg*, etc. it looks at the ebuild, then takes a while to calculate all the dependencies, etc., and then it proceeds to download and build the ebuild(s).

What the heck is the cache used for? What is actually happening when it says "updating portage cache"? Maybe a more important question, does every user need to do this time-consuming step every single time they sync? Maybe there could be a "emerge sync --nocacheupdate" if the "cache" only helps with the searching feature or something... I don't understand what it has to do with getting the new ebuilds and emerging new packages.

If there's documentation somewhere on this, please point it out. I couldn't find it and the portage code isn't documented.

[ciaranm]

The cache is used to speed up dep calculation by about an order of magnitude. When you do emerge -pv blah, it *doesn't* hit the actual ebuilds. Instead, it goes via the metadata cache and gets it from there.

Which is why, if you're really wanting to speed up sync, you're almost certainly better off just syncing the cache and then writing some clever bulk-fetch-on-demand stuff off that. It is not, however, anything like as easy as I make it sound.

Besides, this thread really needs to die. The whole thing was already solved on the gentoo-dev list anyway.

[Genone]

Ideas are nice, implementations are better.

[jmz2]

IMO the speed of syncing is not a problem. If we're supposed to sync no more often than one a day, the speed of the process hardly matters. Besides, using Gentoo requires a bit more knowledge of Linux than the simple point-and-click skills Windows converts have. If people don't want to wait for sync before emerging stuff, what keeps them from using emerge sync && emerge app and leave it running in the background, like they would have to do anyways.

[Vidar]

syncing is not really a problem unless it's the first sync (during installation). That takes forever. Maybe we can have weekly/monthly snapshots of the portage tree in tarball form. When you emerge sync, it scans your timestamp, and it if is older than a week/month, it grabs the tarball instead of rsyncing it. The net effect would be that people that choose to keep their portage tree get the speed of rsync, while you don't spend needless time rsyncing during the first installation.
_________________
"Vidar, Odin's mighty son, he will come to slay the wolf
The sword runs into the heart of Hverdrungs son
So he avenges his father" -- Amon Amarth - Burning Creation

[Chaosite]

[Vidar]

Oh... sorry. I always figured that was something entirely different.
_________________
"Vidar, Odin's mighty son, he will come to slay the wolf
The sword runs into the heart of Hverdrungs son
So he avenges his father" -- Amon Amarth - Burning Creation

[colonel_dolphin]