Emerge sync takes forever, uses 98% CPU

Message

GNUtoo · Post by **GNUtoo** » Sat Dec 17, 2005 1:37 am

roughly:
-what is portage cache???

red-wolf76 · Post by **red-wolf76** » Sat Dec 17, 2005 10:37 am

Well, any sort of cache usually stores remotely available things locally, so you or programs have the luxury of accessing them even when the network is down or currently not available. Your browser caches images for faster repeated loading of often viewed pages, etc.

Portage cache stores the information about every package that is available for your Gentoo (according to architecture and keywords, etc.). Everytime you emerge --sync, it checks your cache against the rsync mirror and downloads the package information that has changes. Hence, there's the notification of a "Speedup" which gives you the time factor you saved by only syncing the relevant parts instead of re-downloading the whole shebang.

But to seamlessly integrate what you just downloaded, Portage needs to check and file that information in - tadaa - the cache and that's what's taking so long because of local syncing commands when KDE packages are concerned.

At least this is how I - a total n00b where inner workings of Gentoo are concerned - understand it. Anyone feel free to correct the blatant errors...

GNUtoo · Post by **GNUtoo** » Sat Dec 17, 2005 2:03 pm

red-wolf76 wrote:Well, any sort of cache usually stores remotely available things locally, so you or programs have the luxury of accessing them even when the network is down or currently not available. Your browser caches images for faster repeated loading of often viewed pages, etc.

Portage cache stores the information about every package that is available for your Gentoo (according to architecture and keywords, etc.). Everytime you emerge --sync, it checks your cache against the rsync mirror and downloads the package information that has changes. Hence, there's the notification of a "Speedup" which gives you the time factor you saved by only syncing the relevant parts instead of re-downloading the whole shebang.

But to seamlessly integrate what you just downloaded, Portage needs to check and file that information in - tadaa - the cache and that's what's taking so long because of local syncing commands when KDE packages are concerned.

At least this is how I - a total n00b where inner workings of Gentoo are concerned - understand it. Anyone feel free to correct the blatant errors...

so a portage tarball snapshot has his cache already updated???
so why it is not done by default???
is it because it's modular and at anytime develloper can summit a portage ebuild?
and so building the cache is impossible (because of concurent ebuild posting etc...)
and so because tarball are freezed state of the portage tree...building the cache is possible for developers and so they've done it

Sprotte · Post by **Sprotte** » Sat Dec 17, 2005 4:58 pm

drescherjm wrote:Maybe I am nuts but I can not believe that people are spending a lot of time hacking portage when the simple solution of using cdb (stated many times in this and other threads) nearly eliminates this problem and it takes possibly 2 minutes to install... emerge metadata for me takes a few seconds, I admit I have not tested this on a P1 200 though...

1. it´s actually very easy to "hack portage"

2. the effect of using cdb is not the same. The metadata part will still take longer than with the above portage fix. So it´s not a "solution".

drescherjm · Post by **drescherjm** » Sat Dec 17, 2005 6:22 pm

it´s actually very easy to "hack portage"

The instructions seemed like a lot of work especially compaired to what it took me to install and use cdb. But I do see there may be some use.

red-wolf76 · Post by **red-wolf76** » Sat Dec 17, 2005 10:49 pm

new_to_non_X86 wrote:so a portage tarball snapshot has his cache already updated???
so why it is not done by default???

Because you might not want to download every single piece of information that is already up-to-date again, and again, and again... It saves bandwidth and even if you're on a T1, it's not really good form to sync every five minutes because other people might be using the 'net too.

is it because it's modular and at anytime develloper can summit a portage ebuild?
and so building the cache is impossible (because of concurent ebuild posting etc...)
and so because tarball are freezed state of the portage tree...building the cache is possible for developers and so they've done it

I've sort of lost you here

Exactly what are you talking about?

nmbrthry · Post by **nmbrthry** » Sun Dec 18, 2005 6:55 am

The wait is finally over...

Portage 2.1-pre is now in the tree. I have been using it all day, currently updating world (which includes 73 packges to get modular X.org up to RC4), and it seems to be fairly stable (for a prerelease). And... it is much faster, in every aspect. This includes the metadata phase of the sync.

So if you are willing to be a little bleeding edge, help Gentoo by installing the new Portage and filing bugs as you find them. (I haven't found any yet.) Oh, it doesn't work with the custom database cache backends. The module system is different now, though I think someone put together a quick fix to get cdb working (see the sticky thread in this forum).

drescherjm · Post by **drescherjm** » Sun Dec 18, 2005 9:20 am

Portage 2.1-pre is now in the tree.

Thanks for the info.

So if you are willing to be a little bleeding edge, help Gentoo by installing the new Portage and filing bugs as you find them.

I hope that its not that bleeding edge that its buggy. I've never had a single problem using the ~ portage in the past.

(I haven't found any yet.) Oh, it doesn't work with the custom database cache backends. The module system is different now, though I think someone put together a quick fix to get cdb working (see the sticky thread in this forum).

Well actually if I did an update world now it would break my system because I use cdb. Thanks again for the warning...

xtaski · Post by **xtaski** » Sun Dec 18, 2005 6:09 pm

nice, very nice; portage screams now

1U · Post by 1U » Mon Jan 09, 2006 7:00 pm

I'm glad this wasn't something just wrong with my system. Sorry for posting a duplicate of this. Had the gentoo forums allowed me to search properly without stripping half of the terms I would have found this thread sooner.

joey_knisch · Post by **joey_knisch** » Tue Jan 10, 2006 4:46 pm

1U wrote:I'm glad this wasn't something just wrong with my system. Sorry for posting a duplicate of this. Had the gentoo forums allowed me to search properly without stripping half of the terms I would have found this thread sooner.

Don't worry. I think about half of us ended up here for that reason. I know I did.

pavkb · Post by **pavkb** » Wed Jan 11, 2006 5:14 am

From the above post, this is fixed in 2.1_pre3_r1, which is masked with ~amd64

Since portage is vital part of the system, what precaution's do i take to upgrade to this version.

A small HOWTO would be nice for novice(s) like me.

Thanks for a great job.

1U · Post by 1U » Wed Jan 11, 2006 5:28 am

Just emerge portage and that's all.

pavkb · Post by **pavkb** » Thu Jan 12, 2006 4:44 am

In case is i build with ACCEPT_KEYWORDS, is there any way i can back out.

Code: Select all

emerge -upv portage

These are the packages that I would merge, in order:

Calculating dependencies ...done!

Total size of downloads: 0 kB

Code: Select all

ACCEPT_KEYWORDS=~amd64 emerge -upv portage

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild     U ] sys-devel/patch-2.5.9-r1 [2.5.9] -build -static 0 kB
[ebuild     U ] sys-apps/sandbox-1.2.17 [1.2.12] 227 kB
[ebuild     U ] app-misc/pax-utils-0.1.6 [0.1.4] -caps 46 kB
[ebuild     U ] sys-apps/debianutils-2.15 [2.14.1-r1] -build -static 181 kB
[ebuild     U ] sys-libs/gpm-1.20.1-r5 [1.20.1-r4] -emacs (-selinux) 7 kB
[ebuild     U ] sys-libs/ncurses-5.5-r1 [5.4-r6] -bootstrap -build -debug +doc +gpm -minimal -nocxx -unicode 2,259 kB
[ebuild     U ] app-shells/bash-3.1_p5 [3.0-r12] (-afs) -bashlogger -build +nls -unicode 2,490 kB
[ebuild     U ] sys-apps/portage-2.1_pre3-r1 [2.0.53] -build +doc (-selinux) 242 kB


Total size of downloads: 5,456 kB

adsmith · Post by **adsmith** » Thu Jan 12, 2006 1:13 pm

no, no, no.. just put "sys-apps/portage ~amd64" into /etc/portage/package.keywords.

there is no reason to do all its deps ~arch, and ACCEPT_KEYWORDS is a bad habit anyway.

drescherjm · Post by **drescherjm** » Thu Jan 12, 2006 2:19 pm

Agreed...

There are a couple of problems when you use ACCEPT_KEYWORDS instead of /etc/portage/package.keywords. A possibly unintended side effect is using ACCEPT_KEYWORDS does ~arch for some of dependencies which may not be whar was intended. Also and if you do an emerge that includes any of these packages there is a chance that portage will downgrade back to the stable version which could break things. This is unless you use the deprecated -U param to portage which may go away in the future.

bombcar · Post by **bombcar** » Thu Jan 26, 2006 3:49 am

I solved this problem by having my webserver be a portage mirror, syncing every day.

And then all my other machines sync to that automatically a few hours later.

ryan42 · Post by **ryan42** » Sun Feb 05, 2006 10:32 pm

I as well have this problem. But when it gets to the cache section it's taking over 2 hours and utilizing upwards of 90% all the time. I am on the PowerPC platform and this is the first emerge --sync I've done. I hope future ones won't take so long.

drescherjm · Post by **drescherjm** » Mon Feb 06, 2006 5:48 pm

I hope future ones won't take so long.

It should not take this long unless you have a very slow computer. What version of portage are you using??

jspeybro · Post by **jspeybro** » Thu Feb 09, 2006 9:49 pm

Well, I have a slow computer...
I'm trying to install gentoo on a toshiba satellite 2520CDS laptop (amd k6 400MHz).
I had to do strange things to actually be able to start installing gentoo: I had to boot with the debian boot-cd, start the debian installation and as soon as the network was configured I switched to a terminal, wiped all the files and downloaded stage3 and the latest portage snapshot.
I am no confronted with the stuck-on-50%-thing. It took more than an hour to get to 51% and even more to get further.
I aborted the emerge --metadata, but now I'm not able to emerge the gentoo-sources. I already tried the workaround to exlclude the KDE-stuff as sugested somewhere in this tread, but that didn't help.

I updated portage (to the masked version with keyword ~x86) but that didn't help much either.
I'm a bit stuck now. I'll let it run overnight and see what happens, but meanwhile, if you guys have any other suggestions, I'd be happy to hear them.

EDIT: It actually took 7 hours to complete the update of the metadata. But it still didn't solve the problem with compilling the kernel, so I'll look fon another answer...

Johan

ferringb · Post by **ferringb** » Fri Feb 10, 2006 8:22 am

red-wolf76 wrote:Everytime you emerge --sync, it checks your cache against the rsync mirror and downloads the package information that has changes. Hence, there's the notification of a "Speedup" which gives you the time factor you saved by only syncing the relevant parts instead of re-downloading the whole shebang.

Nope, that's pretty far off from what occurs actually

(although it is pretty similar to how apt does it's thing).

Portage downloads the tree (rsync'ing). That tree is standalone, it *is* the packages. Each package has certain metadata keys in it that portage needs (DEPEND, SRC_URI, LICENSE fex); sourcing the ebuild to get that information is pretty damn slow however.

Upstream where we generate the rsync image, we also generate the cache for users; that data is distributed in the tree, thus it's locally on your system. The 'updating the cache' is transferring (and mild transformation of) the cache to your local cache, so you don't have to suffer through running without an up to date cache.

Carnildo · Post by **Carnildo** » Wed Feb 15, 2006 6:42 am

ferringb wrote:The 'updating the cache' is transferring (and mild transformation of) the cache to your local cache, so you don't have to suffer through running without an up to date cache.

If all it's doing is transferring data over the internet, then why does it take so damn long? My last "emerge sync" took 45 minutes to "update the cache", and spent most of that time with the CPU pegged at 100%.

Athlon XP2800+
1 GB RAM
80GB 7200RPM hard drive
4Mbit down/384kbit up

spamspam · Post by **spamspam** » Tue Feb 21, 2006 12:51 am

Carnildo wrote:
ferringb wrote:The 'updating the cache' is transferring (and mild transformation of) the cache to your local cache, so you don't have to suffer through running without an up to date cache.
If all it's doing is transferring data over the internet, then why does it take so damn long? My last "emerge sync" took 45 minutes to "update the cache", and spent most of that time with the CPU pegged at 100%.

As stated in the thread, it's not the downloading that takes a long time, it's the metadata updating. Every time anything changes ((# packages) * (#packages) * (#packages) times), it deletes the file and re-writes it. You can fix this by either upgrading to the unstable version of portage, or removing every sync() (NOT sync(somestuff)) call in your copy of /usr/lib/portage/pym/portage.py .

Sachankara · Post by **Sachankara** » Thu Feb 23, 2006 3:12 pm

new_to_non_X86 wrote:so a portage tarball snapshot has his cache already updated???
so why it is not done by default???

Actually, the portage cache is stored under /var/cache/edb/dep/, so if you want to sync many computers over your own network, just compress /usr/portage/ plus /var/cache/edb/dep/ and send them to the other computers. You'll save hours of processing time.

_pF_ · Post by **_pF_** » Tue Mar 14, 2006 5:23 am

I suffer from the portage emerge sync bug, where updating the portage cache sticks at about 50% to 51% for an hour or more.

About a month ago this problem disappeared, and the entire sync completed in about eight minutes. This was a big sync, resulting in a few days' continuous compilation.

Today, though, emerge sync is again taking hours stuck halfway during the cache update.

One cpu is running at 99%.

Was there a recent release of portage that fixed the bug, then got broken again?

Emerge sync takes forever, uses 98% CPU

Very Strange - portage 2.0.54