Emerge sync takes forever, uses 98% CPU

Message

rk_cr · Post by **rk_cr** » Tue Oct 04, 2005 5:58 am

I hate to say this but after reading this and testing a sync again it's back...

Damn you KDE. I thought I got rid of you years ago.

lost+found · Post by **lost+found** » Tue Oct 04, 2005 6:21 am

Maybe the problem becomes worse, as there will be more and more KDE split ebuilds, and the KDE monolithic ebuilds will continue until 4.0...

Gnome people:
/etc/make.conf

Code: Select all

RSYNC_EXCLUDEFROM="/etc/portage/rsync_excludes"

/etc/portage/rsync_excludes

Code: Select all

- kde-*/
- metadata/cache/kde-*/

...and to prevent emerge --metadata to calculate the cache again with the outdated stuff: remove /usr/portage/kde-*/ and /usr/portage/metadata/cache/kde-*/ manually.

Sachankara · Post by **Sachankara** » Tue Oct 04, 2005 12:39 pm

Yep, the problem is back. *lol* :/

real 13m31.972s
user 9m54.254s
sys 0m30.140s

widremann · Post by **widremann** » Tue Oct 04, 2005 1:18 pm

It never went away for me, but I only sync once every two weeks or so. There may have been a period in between there when it went away. It takes upwards of 20 minutes now to complete the whole sync operation.

steves · Post by **steves** » Tue Oct 04, 2005 10:15 pm

I had a similar problem and used the new command eclean to clean up distfiles and packages. It may have been just chance but the next updates ran as normal and no delay through 50% and far shorter time.

You need to use the latest version of gentoolkit though.

Deep-VI · Post by **Deep-VI** » Thu Oct 06, 2005 4:01 pm

Ahh, it's much better now as of portage-2.0.51.22-r3. The speed is back!

stef · Post by **stef** » Fri Oct 14, 2005 11:16 am

yes, on my k6 300 updating the portage cache takes way > 30 min and stays long time at 50 %
(no binary packages - no /usr/portage/packages dir)
My system has enough free memory, so no swap is used...
some times ago it only took a few minutes ... so performance seems to get worse (or it's just because of bigger portage tree)

toocO.ol · Post by **toocO.ol** » Fri Oct 14, 2005 2:56 pm

hi I agree that this is a real problem which devs don't want to face or answer angrily that your are doing something stupid and it's not PORTAGE faults. And it's pretty much Portage/Gentoo fault. Why should any user have to do outside maneouvres in order to fix this issue? No other distro has this problem when I used Debian their package manager was such a breeze even on damn ol' putter. In comparison Portage spends ages SCREECHING the hard drive to do emerge -s foo and emerge --syn and emerge -whatever.

So since emerge and portage are such a unusable piece of crap software the only choice to fix this issue is simply to switch to Eix and Portage cdb. Or to migrate to FreeBSD as apparently many are doing due to this issue.

kev009 · Post by **kev009** » Mon Oct 17, 2005 5:41 am

I too was experiancing the 50% hang up. I cannot say with absolute certainty as I did not perform the tasks atomically, but I sped my horridly slow (40minute!!!) syncs to roughly 3 minutes. Firstly, I wiped /usr/portage and /var/cache/edb. I then reinstalled a portage snapshot and synced but the sync improved only marginally (maybe a minute or two). This tells me a few things, first although fragmentation plays a role in the speed of syncs, it is not the root problem we are experiancing. Next I proceeded to update Python to 2.4.2 and run python-cleaner. My syncs were instantly an order of magnitude faster! I ask anyone experiancing this issue try my method atomically (IE don't do the disk wipe first), and post results to confirm or deny this fix.

alinv · Post by **alinv** » Mon Oct 17, 2005 7:23 am

Sounds interesting.
What is this python-cleaner? I can't find any reference of it.

schnake · Post by **schnake** » Mon Oct 17, 2005 8:29 am

Here is what I found searching the net for "python cleaner":

http://www.appealinggifts.com/cord-cord ... eaner.html

LOL

radulucian · Post by **radulucian** » Mon Oct 17, 2005 12:26 pm

did you mean python-updater ?

ethrandil · Post by **ethrandil** » Mon Oct 17, 2005 11:30 pm

Hi,
I tried to get better results with an loopback-device and let me say: I got them!

I created a 3G big file with 'dd if=/dev/zero of=portage-file bs=1K count=3M' and formated it with reiserfs.
Then I mounted it ('mount portage-file portage-file-mount -t reiserfs -o loop') copied /usr/portage onto it, and made /usr/portage a symlink to the new filesystem.

I ran 'emerge --metadata' in:
real 2m33.763s
user 0m23.443s
sys 0m7.403s

and had no noticeable hang at 50%.
The speed before was... i did't want to test it again because it was definitly TOO LONG! about > 15 min!

So I guess this could be a fragmentation problem. Maybe I'll bench different Filesystems on the loopfile later.

- Eth

alinv · Post by **alinv** » Tue Oct 18, 2005 12:48 am

Using anydbm as a portage backend seems to speed up things. Now, emerge --metadata takes about 3 minutes to complete, w/o any slowdown around 50%.
But I noticed another strange thing: the first emerge -pu world after updating cache takes about 10 minutes, while subsequent runs take less than a minute.
I don't get it

DrZoidberg · Post by **DrZoidberg** » Tue Oct 18, 2005 1:08 am

I've had this problem for over a year running on PPC. I have had very slow "Updating Portage cache" with the hang at 50% on my 500MHz G4 pismo powerbook. I thought it was just because my laptop was old. Then I had the same problem on my G4 1.2GHz iBook and on a 3GHz pentium 4 desktop!

aethyr · Post by **aethyr** » Tue Oct 18, 2005 1:59 am

nevermind, again:
http://forums.gentoo.org/viewtopic-p-28 ... ml#2815795

kev009 · Post by **kev009** » Tue Oct 18, 2005 4:39 am

radulucian wrote:did you mean python-updater ?

Yes, sorry folks.. getting perl and python programs confused

.

mrv · Post by **mrv** » Tue Oct 18, 2005 4:01 pm

I've also had this issue for 1-2 weeks with a stable system (only couple of ~amd64 packages). I'm running sys-apps/portage-2.0.51.22-r3 and dev-lang/python-2.4.1-r1. The cache updating phase of emerge --sync sits a very long time at 52 %.

-mrv-

dalek · Post by **dalek** » Tue Oct 18, 2005 4:48 pm

Well I have three rigs and have noticed the same thing.

Rig one is in the sig and has been running for a while, since about my join date here. I did copy the files, including distfiles, to a seperate partition, delete the old, then copy it back again. /usr/portage is on it's own partition by the way. We'll see if that defrags any and helps. reiserfs all the way around too.

Rig two is a server setup that runs folding and has /boot and / and that is all. Same thing at about 50% though. It is a 800MHz rig with 128MBs of ram.

Rig three is a Compaq Proliant 6000 with quad CPUs and 128MBs of ram set up as a server. This is a very recent install and it was slow from the get go. It hung a bit at 50% after the first sync. It to has /boot and / and that's it.

Rigs two and three runs folding and that is about it. If someone wants me too, I will post the make.conf and/or emerge info if you need it. Just let me know.

Since I am on dial-up, I have noticed that it takes longer to download all the files too. I think when I first started using Gentoo, there was about 60 or 70,000 files, there was about 128,000 last night. Seems like that may have something to do with it but I'm not etching that in stone either.

Still LOVE my Gentoo though. I do wish I had DSL though. Of course, I have been dating a lady in Mobile and may move there, they will have DSL there I'm sure. I suspect that about two weeks after I move, they will send me a email that DSL is available here.

They put A/C in our schools the year after I graduated too.

That's my luck in life. I'm one in a few thousand with my skin disorder too. If it wasn't for bad luck, I would have no luck at all. My lady is a silver lining though. She is super sweet.

I need to loose weight and she needs to gain some. She only weighs 97 lbs.

I LOVE every ounce though. I have to, there aren't many ounces there.

Later

shimbob · Post by **shimbob** » Wed Oct 19, 2005 4:42 am

4 machines here showing the same problem. 50 and 51% take minutes to go through, but it speeds along fine before and after those percentages. emerge is hogging 99% cpu

It's a recent thing, within the past 2-3 weeks and came rather sudden.

Reiserv4 on all machines, one 733mhz P3, two 3ghz P4 and one smp amd 2400+ MP machine.

StringCheesian · Post by **StringCheesian** » Wed Oct 19, 2005 4:57 am

Same problem on reiserfs 3.6

sys-apps/portage-2.0.53_rc5
net-misc/rsync-2.6.6
sys-kernel/gentoo-sources-2.6.13-r3
In /etc/make.conf: SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
Athlon XP 3200 with 512 MB ram, no other significant processes besides KDE, Konsole, and Firefox

kev009 · Post by **kev009** » Wed Oct 19, 2005 5:33 am

Well, the symtoms are now appearing on my Opteron 248 with a U320 10k RPM disk! Although not as drastic as some of my other machines, I guess none are immune.

mrv · Post by **mrv** » Wed Oct 19, 2005 7:57 am

I don't know if this is a "solution", but I ran /usr/sbin/python-updater and after that "emerge metadata" no longer took so long (didn't stuck at 52 %).

Let's see if the situation is still same tomorrow...

EDIT: Ok, it didn't work. "emerge --sync" still stucks at 52 % and takes the whole CPU time...

-mrv-

marsclic · Post by **marsclic** » Wed Oct 19, 2005 8:30 am

I have an athlon64 and a Reiser4 fs on a 4 disk raid-0, and the emerge sync process was taking about 5 minutes, much much longer than it used to be. It also hanged for a long time at the 50% mark, without any disk or network activity. I ran python-updater, and emerge --metadata took very little time to complete. So to confirm, I did a "time emerge sync" again, and this is what I got:

Code: Select all

deleting x11-libs/gtk+/files/digest-gtk+-2.6.7

Number of files: 127953
Number of files transferred: 50
Total file size: 101973845 bytes
Total transferred file size: 156661 bytes
Literal data: 156661 bytes
Matched data: 0 bytes
File list size: 3045801
Total bytes sent: 1149
Total bytes received: 3101344

sent 1149 bytes  received 3101344 bytes  74758.87 bytes/sec
total size is 101973845  speedup is 32.87

>>> Updating Portage cache:  100%

real    1m3.832s
user    0m8.826s
sys     0m4.950s

This now is substantially faster. Many thanks for who suggested this fix.

salivian · Post by **salivian** » Thu Oct 20, 2005 12:45 am

I think testing emerge metadata or sync, after python-updater and not too long after a previous sync may not be appropriate, because the cache got recently updated, there is a limited amount of files to make portage do the work.

try removing /var/cache/edb/ and run emerge metadata, this really cause all the cache to be rebuilt. In my experience, I am sorry to say it will still stuck at 50 - 52%.

Emerge sync takes forever, uses 98% CPU

most retarded thing about Gentoo

python-updater trick worked for me