Emerge sync takes forever, uses 98% CPU

xtaski · Last edited by xtaski on Sat Sep 24, 2005 5:25 pm; edited 2 times in total

Not sure what happened all the sudden but running emerge --sync uses up 98% CPU and takes forever. It used to take a couple minute to update - now it's like 20-30 mins... It's slow on the "Updating Portage Cache... .X%" - The rsync is fine...

Any ideas? I looked through the forums and didn't see anything like this... it just started this last week...

I sync about every day or two and am updated on all packages.... can't figure out what's doing this.

NeddySeagoon · Posted: Fri Sep 23, 2005 5:34 pm Post subject:

xtaski,

Do you have a lot of binary packages?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

xtaski · Posted: Fri Sep 23, 2005 5:42 pm Post subject:

nope; only binary is probably openoffice and nvidia... everything else is built from source...

ldoe3000

Anyone else experiencing really long "Updating Portage cache:" times when doing an emerge sync? Even if I do an "emerge metadata", it still takes about 3-4 minutes on that step alone. In the past it took under 1 minute.. Is it just me?
_________________
Athlon64 3500+ MSI Neo2 Platinum
2X74 WD Raptor Raid 0
2X512MB Crucial Ballistix
ATI x800XT PE (8.18.6)
SB Audigy 2 ZS
Dell 24" Widescreen LCD
Dual boot: Gentoo x86_64 2.6.13-r4 , Windows XP x64

cantao · Apprentice Joined: 07 Jan 2004 Posts: 154

Yep, same behavior here... I thought it was just my imagination

Any clues?

Cheers, Cantão!

ldoe3000 · Posted: Fri Sep 23, 2005 9:12 pm Post subject:

None, except we're up to 126000 files.. Perhaps Portage has hit some threshold where it can't really keep up with the volume now.. Total guess there.
_________________
Athlon64 3500+ MSI Neo2 Platinum
2X74 WD Raptor Raid 0
2X512MB Crucial Ballistix
ATI x800XT PE (8.18.6)
SB Audigy 2 ZS
Dell 24" Widescreen LCD
Dual boot: Gentoo x86_64 2.6.13-r4 , Windows XP x64

satanskin · Guru Joined: 24 Apr 2005 Posts: 353

They should switch it over and use some sort of database with it. Maybe support one of the C++ versions of portage...

Regardless, I'm having the same thing. Takes a while.

barum87 · n00b Joined: 28 Mar 2005 Posts: 62

Me too.
I just thought that my system wasn't optimized enough, but I guess it's not just my problem.

tam · Guru Joined: 04 Mar 2003 Posts: 569 Location: freiburg.de

In deed, portage should become faster.
_________________
http://tam.belchenstuermer.de/

toralf · Posted: Sat Sep 24, 2005 3:00 am Post subject:

An "emerge sync" is since some days/weeks significantly slower than before. I see "Updating Portage cache: 50%" for 1-2 minute(s) or longer before it changes.
This is independent from the general question how the process can be speed up.

NeddySeagoon · Posted: Sat Sep 24, 2005 3:26 am Post subject:

xtaski,

Sorry, I wasn't clear.

Do you have

Gentii · Guru Joined: 01 Feb 2004 Posts: 306

Yep, I have exactly the same problem. In fact I just asked on gentoo chan why this step was getting so slow.. Anyone told me to give cdb module a try ( http://forums.gentoo.org/viewtopic-t-261580-highlight-cdb+module.html ).
So what's the best workaround (easiest and cleanest to set up) ? cdb , one c++ portage like sportage, or something else?

Btw, this post has nothing to do with amd64, maybe it should be moved

toralf · Posted: Sat Sep 24, 2005 6:55 am Post subject:

I have a similar problem, after 50% is completed emerge uses full cpu power but that's all for a while http://forums.gentoo.org/viewtopic.php?p=2749780#2749780
BTW here is my config

xtaski · Posted: Sat Sep 24, 2005 12:33 pm Post subject:

kmare · Posted: Sat Sep 24, 2005 12:48 pm Post subject:

same problem here..

xtaski · Posted: Sat Sep 24, 2005 1:19 pm Post subject:

Not sure if this helps, but here's the summary after a --sync:

NeddySeagoon · Posted: Sat Sep 24, 2005 1:50 pm Post subject:

xtaski,

If you never use the binary packages, you can turn off FEATURES="buildpkg" and get rid of the /usr/portage/packages directory.
emerge sync will not to the binary package bit at the end.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

widremann · Veteran Joined: 14 Mar 2005 Posts: 1149

I have the same problem and I don't have "buildpkg" set. It just takes forever and the CPU has high utilization. I don't even have a /usr/portage/packages directory, either.

Nijinski · Posted: Sat Sep 24, 2005 3:59 pm Post subject:

Same problem here, not limited to AMD64 either, getting it on SPARC and x86 as well.

Big pause at 50% then starts off again.
_________________
Help ! Help ! I'm being repressed

xtaski · Posted: Sat Sep 24, 2005 4:53 pm Post subject:

Thanks Neddy! That fixed my problem - guess I never really knew why I kept including that feature... now I know not to.

cmtaylor · n00b Joined: 09 Aug 2004 Posts: 36 Location: Houghton, MI

i'm having the same problem as wildremann -- i don't have buildpkg set, and i don't have the /use/portage/packages directory. any other possibilities?

xtaski · Posted: Sat Sep 24, 2005 5:25 pm Post subject:

Can you post your make.conf? Maybe there's something else in there...

The Mountain Man · l33t Joined: 02 Sep 2003 Posts: 630

Glad I'm not alone. Takes some minutes and uses almost 100% of my CPU. Wonder what's up?
_________________
I suck at signatures.

i92guboj

The tittle says it all: my box can spend 10-20 minutes (I havent meassured exactly, but is noticeably slower then before) to retrieve all the metadata, after a sync. I have experienced this problem since a couple of months or so. The filesystem is ext2, that is fast, and I checked that is 100% error free. The cpu is mostly idle when doing that, and there is enough free memory and no swap at all when performing this operation.

I noticed that the number of files in portage has grown in the last months a lot, but I dont know if that is what is causing this odd thing. I use a couple of overlays. One in /usr/local/portage with 10 or 15 ebuilds only in it. The other is bigger, in /usr/local/gentopia, with the ebuild from here: http://gentopia.gentooexperimental.org./wiki

Has anyone experienced this slowdown in the last months, or is it just me? It worries my, since, if the portage size is the real problem this could become a serious problem in the next few months...

_________________
Formerly known as 6thpink.
Gentoo Handbook | My home | My rock band

lost+found · Posted: Sun Sep 25, 2005 11:03 am Post subject:

I think it's a filesystem problem: inefficient storage of small files, spread over the hd everywhere...
Some time ago, i put back a Portage snapshot from a tar.bz2, after deleting /usr/portage... after that it was fast again... Maybe ReiserFS would do it better (it's fast on small files)? It should be used for /usr/portage and /var/cache/edb then. I'm using JFS, so i decided to cut down the Portage tree with rsync_excludes, very fast, but you will have to do some manual work when emerging new packages.