Emerge sync takes forever, uses 98% CPU

Message

xtaski · Post by **xtaski** » Fri Sep 23, 2005 10:23 pm

Not sure what happened all the sudden but running emerge --sync uses up 98% CPU and takes forever. It used to take a couple minute to update - now it's like 20-30 mins... It's slow on the "Updating Portage Cache... .X%" - The rsync is fine...

Any ideas? I looked through the forums and didn't see anything like this... it just started this last week...

I sync about every day or two and am updated on all packages.... can't figure out what's doing this.

Post by **NeddySeagoon** » Fri Sep 23, 2005 10:34 pm

xtaski,

Do you have a lot of binary packages?

xtaski · Post by **xtaski** » Fri Sep 23, 2005 10:42 pm

nope; only binary is probably openoffice and nvidia... everything else is built from source...

ldoe3000 · Post by **ldoe3000** » Sat Sep 24, 2005 1:18 am

Anyone else experiencing really long "Updating Portage cache:" times when doing an emerge sync? Even if I do an "emerge metadata", it still takes about 3-4 minutes on that step alone. In the past it took under 1 minute.. Is it just me?

cantao · Post by **cantao** » Sat Sep 24, 2005 1:26 am

Yep, same behavior here... I thought it was just my imagination

Any clues?

Cheers, Cantão!

ldoe3000 · Post by **ldoe3000** » Sat Sep 24, 2005 2:12 am

None, except we're up to 126000 files.. Perhaps Portage has hit some threshold where it can't really keep up with the volume now.. Total guess there.

satanskin · Post by **satanskin** » Sat Sep 24, 2005 3:02 am

They should switch it over and use some sort of database with it. Maybe support one of the C++ versions of portage...

Regardless, I'm having the same thing. Takes a while.

barum87 · Post by **barum87** » Sat Sep 24, 2005 3:18 am

Me too.
I just thought that my system wasn't optimized enough, but I guess it's not just my problem.

tam · Post by **tam** » Sat Sep 24, 2005 7:52 am

In deed, portage should become faster.

Post by **toralf** » Sat Sep 24, 2005 8:00 am

An "emerge sync" is since some days/weeks significantly slower than before. I see "Updating Portage cache: 50%" for 1-2 minute(s) or longer before it changes.
This is independent from the general question how the process can be speed up.

Post by **NeddySeagoon** » Sat Sep 24, 2005 8:26 am

xtaski,

Sorry, I wasn't clear.

Do you have

Code: Select all

FEATURES="buildpkg"

set in /etc/make.conf, even by defualt ?
You need not have installed any binary packages. Look in

Code: Select all

emerge --info

since that will show you things which are on by defualt too.

This makes portage save a binary of everything you build, at the end of emerge sync it does a lot checking of these binaries.
They are saved in /usr/portage/packages by defualt.

Gentii · Post by **Gentii** » Sat Sep 24, 2005 11:30 am

Yep, I have exactly the same problem. In fact I just asked on gentoo chan why this step was getting so slow.. Anyone told me to give cdb module a try ( http://forums.gentoo.org/viewtopic-t-26 ... odule.html ).
So what's the best workaround (easiest and cleanest to set up) ? cdb , one c++ portage like sportage, or something else?

Btw, this post has nothing to do with amd64, maybe it should be moved

Post by **toralf** » Sat Sep 24, 2005 11:55 am

I have a similar problem, after 50% is completed emerge uses full cpu power but that's all for a while http://forums.gentoo.org/viewtopic.php? ... 80#2749780
BTW here is my config

Code: Select all

CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"

USE="acpi aim apache2 browserplugin cardbus cddb cdparanoia dvd dvdr dvdread
     ethereal font-server innodb jai javascript jimi live mbox mmx
     mmxext mng nagios-dns nagios-ntp nagios-ping nagios-s network
     no-old-linux nsplugin pcmcia real slp smime sse sse2 subversion svg
     usb userlocales webdav wifi xanim xfs -apm -gnome"
LINGUAS="de en"

MAKEOPTS="-j2"
PORTAGE_NICENESS="3"
PORT_LOGDIR="/var/log/portage"
FEATURES="ccache"
CCACHE_SIZE="1G"

xtaski · Post by **xtaski** » Sat Sep 24, 2005 5:33 pm

Do you have
Code: Select all
FEATURES="buildpkg"
set in /etc/make.conf, even by defualt ?
You need not have installed any binary packages. Look in
Code: Select all
emerge --info
since that will show you things which are on by defualt too.

This makes portage save a binary of everything you build, at the end of emerge sync it does a lot checking of these binaries.
They are saved in /usr/portage/packages by defualt.

Ok, yes I do have "buildpkg" - how do I "fix" this then?

kmare · Post by **kmare** » Sat Sep 24, 2005 5:48 pm

same problem here..

xtaski · Post by **xtaski** » Sat Sep 24, 2005 6:19 pm

Not sure if this helps, but here's the summary after a --sync:

Code: Select all

Number of files: 125563
Number of files transferred: 1198
Total file size: 99513410 bytes
Total transferred file size: 2450380 bytes
Literal data: 2450380 bytes
Matched data: 0 bytes
File list size: 2862533
File list generation time: 15.113 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33240
Total bytes received: 5376530

sent 33240 bytes  received 5376530 bytes  53298.23 bytes/sec
total size is 99513410  speedup is 18.40

>>> Updating Portage cache:  100%

Post by **NeddySeagoon** » Sat Sep 24, 2005 6:50 pm

xtaski,

If you never use the binary packages, you can turn off FEATURES="buildpkg" and get rid of the /usr/portage/packages directory.
emerge sync will not to the binary package bit at the end.

widremann · Post by **widremann** » Sat Sep 24, 2005 8:33 pm

I have the same problem and I don't have "buildpkg" set. It just takes forever and the CPU has high utilization. I don't even have a /usr/portage/packages directory, either.

Nijinski · Post by **Nijinski** » Sat Sep 24, 2005 8:59 pm

Same problem here, not limited to AMD64 either, getting it on SPARC and x86 as well.

Big pause at 50% then starts off again.

xtaski · Post by **xtaski** » Sat Sep 24, 2005 9:53 pm

Thanks Neddy! That fixed my problem - guess I never really knew why I kept including that feature... now I know not to.

cmtaylor · Post by **cmtaylor** » Sat Sep 24, 2005 10:24 pm

i'm having the same problem as wildremann -- i don't have buildpkg set, and i don't have the /use/portage/packages directory. any other possibilities?

xtaski · Post by **xtaski** » Sat Sep 24, 2005 10:25 pm

Can you post your make.conf? Maybe there's something else in there...

The Mountain Man · Post by **The Mountain Man** » Sun Sep 25, 2005 2:58 am

Glad I'm not alone. Takes some minutes and uses almost 100% of my CPU. Wonder what's up?

i92guboj · Post by **i92guboj** » Sun Sep 25, 2005 3:11 pm

The tittle says it all: my box can spend 10-20 minutes (I havent meassured exactly, but is noticeably slower then before) to retrieve all the metadata, after a sync. I have experienced this problem since a couple of months or so. The filesystem is ext2, that is fast, and I checked that is 100% error free. The cpu is mostly idle when doing that, and there is enough free memory and no swap at all when performing this operation.

I noticed that the number of files in portage has grown in the last months a lot, but I dont know if that is what is causing this odd thing. I use a couple of overlays. One in /usr/local/portage with 10 or 15 ebuilds only in it. The other is bigger, in /usr/local/gentopia, with the ebuild from here: http://gentopia.gentooexperimental.org./wiki

Has anyone experienced this slowdown in the last months, or is it just me? It worries my, since, if the portage size is the real problem this could become a serious problem in the next few months...

lost+found · Post by **lost+found** » Sun Sep 25, 2005 4:03 pm

I think it's a filesystem problem: inefficient storage of small files, spread over the hd everywhere...
Some time ago, i put back a Portage snapshot from a tar.bz2, after deleting /usr/portage... after that it was fast again... Maybe ReiserFS would do it better (it's fast on small files)? It should be used for /usr/portage and /var/cache/edb then. I'm using JFS, so i decided to cut down the Portage tree with rsync_excludes, very fast, but you will have to do some manual work when emerging new packages.

Emerge sync takes forever, uses 98% CPU

Emerge sync takes forever, uses 98% CPU

Updating Portage cache taking a long time lately

Same here...

slow portage

[Slow Portage] - Retrieving metadata - (Open)