Emerge sync takes forever, uses 98% CPU

rk_cr · n00b Joined: 13 Jan 2004 Posts: 12

I hate to say this but after reading this and testing a sync again it's back...

Damn you KDE. I thought I got rid of you years ago.

lost+found · Posted: Tue Oct 04, 2005 1:21 am Post subject:

Maybe the problem becomes worse, as there will be more and more KDE split ebuilds, and the KDE monolithic ebuilds will continue until 4.0... :-(

Gnome people:
/etc/make.conf

Sachankara · Posted: Tue Oct 04, 2005 7:39 am Post subject:

Yep, the problem is back. *lol* :/

real 13m31.972s
user 9m54.254s
sys 0m30.140s
_________________
Gentoo Hardened Linux 2.6.21 + svorak (Swedish dvorak)

widremann · Veteran Joined: 14 Mar 2005 Posts: 1149

It never went away for me, but I only sync once every two weeks or so. There may have been a period in between there when it went away. It takes upwards of 20 minutes now to complete the whole sync operation.

steves · n00b Joined: 10 Jan 2005 Posts: 30

I had a similar problem and used the new command eclean to clean up distfiles and packages. It may have been just chance but the next updates ran as normal and no delay through 50% and far shorter time.

You need to use the latest version of gentoolkit though.

Deep-VI · n00b Joined: 09 Jan 2005 Posts: 18

Ahh, it's much better now as of portage-2.0.51.22-r3. The speed is back!
_________________
Gentoo/GNU Linux: Choice is power - have it YOUR way.

stef · Tux's lil' helper Joined: 23 Jul 2003 Posts: 90

yes, on my k6 300 updating the portage cache takes way > 30 min and stays long time at 50 %
(no binary packages - no /usr/portage/packages dir)
My system has enough free memory, so no swap is used...
some times ago it only took a few minutes ... so performance seems to get worse (or it's just because of bigger portage tree)

toocO.ol · n00b Joined: 06 Oct 2005 Posts: 16

hi I agree that this is a real problem which devs don't want to face or answer angrily that your are doing something stupid and it's not PORTAGE faults. And it's pretty much Portage/Gentoo fault. Why should any user have to do outside maneouvres in order to fix this issue? No other distro has this problem when I used Debian their package manager was such a breeze even on damn ol' putter. In comparison Portage spends ages SCREECHING the hard drive to do emerge -s foo and emerge --syn and emerge -whatever.

So since emerge and portage are such a unusable piece of crap software the only choice to fix this issue is simply to switch to Eix and Portage cdb. Or to migrate to FreeBSD as apparently many are doing due to this issue.

:-(

kev009 · Posted: Mon Oct 17, 2005 12:41 am Post subject:

I too was experiancing the 50% hang up. I cannot say with absolute certainty as I did not perform the tasks atomically, but I sped my horridly slow (40minute!!!) syncs to roughly 3 minutes. Firstly, I wiped /usr/portage and /var/cache/edb. I then reinstalled a portage snapshot and synced but the sync improved only marginally (maybe a minute or two). This tells me a few things, first although fragmentation plays a role in the speed of syncs, it is not the root problem we are experiancing. Next I proceeded to update Python to 2.4.2 and run python-cleaner. My syncs were instantly an order of magnitude faster! I ask anyone experiancing this issue try my method atomically (IE don't do the disk wipe first), and post results to confirm or deny this fix.
_________________
http://www.kev009.com

alinv · Guru Joined: 19 Nov 2002 Posts: 395 Location: Bucharest

Sounds interesting.
What is this python-cleaner? I can't find any reference of it.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B.

schnake · Posted: Mon Oct 17, 2005 3:29 am Post subject:

Here is what I found searching the net for "python cleaner":

http://www.appealinggifts.com/cord-cordless-python-convertible-vacuum-cleaner.html

LOL

radulucian · Posted: Mon Oct 17, 2005 7:26 am Post subject:

did you mean python-updater ?

ethrandil · n00b Joined: 19 Oct 2003 Posts: 20

Hi,
I tried to get better results with an loopback-device and let me say: I got them!

I created a 3G big file with 'dd if=/dev/zero of=portage-file bs=1K count=3M' and formated it with reiserfs.
Then I mounted it ('mount portage-file portage-file-mount -t reiserfs -o loop') copied /usr/portage onto it, and made /usr/portage a symlink to the new filesystem.

I ran 'emerge --metadata' in:
real 2m33.763s
user 0m23.443s
sys 0m7.403s

and had no noticeable hang at 50%.
The speed before was... i did't want to test it again because it was definitly TOO LONG! about > 15 min!

So I guess this could be a fragmentation problem. Maybe I'll bench different Filesystems on the loopfile later.

- Eth

alinv · Guru Joined: 19 Nov 2002 Posts: 395 Location: Bucharest

Using anydbm as a portage backend seems to speed up things. Now, emerge --metadata takes about 3 minutes to complete, w/o any slowdown around 50%.
But I noticed another strange thing: the first emerge -pu world after updating cache takes about 10 minutes, while subsequent runs take less than a minute.
I don't get it

_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B.

DrZoidberg · Posted: Mon Oct 17, 2005 8:08 pm Post subject:

I've had this problem for over a year running on PPC. I have had very slow "Updating Portage cache" with the hang at 50% on my 500MHz G4 pismo powerbook. I thought it was just because my laptop was old. Then I had the same problem on my G4 1.2GHz iBook and on a 3GHz pentium 4 desktop!

aethyr · Veteran Joined: 06 Apr 2003 Posts: 1085 Location: NYC

nevermind, again:
http://forums.gentoo.org/viewtopic-p-2815795.html#2815795

kev009 · Posted: Mon Oct 17, 2005 11:39 pm Post subject:

mrv · Posted: Tue Oct 18, 2005 11:01 am Post subject:

I've also had this issue for 1-2 weeks with a stable system (only couple of ~amd64 packages). I'm running sys-apps/portage-2.0.51.22-r3 and dev-lang/python-2.4.1-r1. The cache updating phase of emerge --sync sits a very long time at 52 %.

-mrv-

dalek · Posted: Tue Oct 18, 2005 11:48 am Post subject:

Well I have three rigs and have noticed the same thing.

Rig one is in the sig and has been running for a while, since about my join date here. I did copy the files, including distfiles, to a seperate partition, delete the old, then copy it back again. /usr/portage is on it's own partition by the way. We'll see if that defrags any and helps. reiserfs all the way around too.

Rig two is a server setup that runs folding and has /boot and / and that is all. Same thing at about 50% though. It is a 800MHz rig with 128MBs of ram.

Rig three is a Compaq Proliant 6000 with quad CPUs and 128MBs of ram set up as a server. This is a very recent install and it was slow from the get go. It hung a bit at 50% after the first sync. It to has /boot and / and that's it.

Rigs two and three runs folding and that is about it. If someone wants me too, I will post the make.conf and/or emerge info if you need it. Just let me know.

Since I am on dial-up, I have noticed that it takes longer to download all the files too. I think when I first started using Gentoo, there was about 60 or 70,000 files, there was about 128,000 last night. Seems like that may have something to do with it but I'm not etching that in stone either.

Still LOVE my Gentoo though. I do wish I had DSL though. Of course, I have been dating a lady in Mobile and may move there, they will have DSL there I'm sure. I suspect that about two weeks after I move, they will send me a email that DSL is available here. :roll:

They put A/C in our schools the year after I graduated too.

That's my luck in life. I'm one in a few thousand with my skin disorder too. If it wasn't for bad luck, I would have no luck at all. My lady is a silver lining though. She is super sweet.

I need to loose weight and she needs to gain some. She only weighs 97 lbs.

I LOVE every ounce though. I have to, there aren't many ounces there. :oops:

Later

_________________
My rig: ABIT NF7 mobo with sound
AMD 2500+ || Volcano 12+ Cooler
1Gb Kingston Ram || fx 5200 NVIDIA 128Mb video card
Maxtor 80 GB & Western Digital 80GB HD.

Myspace
It's public again too.

shimbob · n00b Joined: 13 Sep 2003 Posts: 30

4 machines here showing the same problem. 50 and 51% take minutes to go through, but it speeds along fine before and after those percentages. emerge is hogging 99% cpu

It's a recent thing, within the past 2-3 weeks and came rather sudden.

Reiserv4 on all machines, one 733mhz P3, two 3ghz P4 and one smp amd 2400+ MP machine.

StringCheesian · l33t Joined: 21 Oct 2003 Posts: 802

Same problem on reiserfs 3.6

sys-apps/portage-2.0.53_rc5
net-misc/rsync-2.6.6
sys-kernel/gentoo-sources-2.6.13-r3
In /etc/make.conf: SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
Athlon XP 3200 with 512 MB ram, no other significant processes besides KDE, Konsole, and Firefox

kev009 · Posted: Wed Oct 19, 2005 12:33 am Post subject:

Well, the symtoms are now appearing on my Opteron 248 with a U320 10k RPM disk! Although not as drastic as some of my other machines, I guess none are immune.
_________________
http://www.kev009.com

mrv · Posted: Wed Oct 19, 2005 2:57 am Post subject:

I don't know if this is a "solution", but I ran /usr/sbin/python-updater and after that "emerge metadata" no longer took so long (didn't stuck at 52 %).

Let's see if the situation is still same tomorrow...

EDIT: Ok, it didn't work. "emerge --sync" still stucks at 52 % and takes the whole CPU time...

-mrv-

marsclic · n00b Joined: 02 Mar 2003 Posts: 67

I have an athlon64 and a Reiser4 fs on a 4 disk raid-0, and the emerge sync process was taking about 5 minutes, much much longer than it used to be. It also hanged for a long time at the 50% mark, without any disk or network activity. I ran python-updater, and emerge --metadata took very little time to complete. So to confirm, I did a "time emerge sync" again, and this is what I got:

salivian · Tux's lil' helper Joined: 15 Sep 2002 Posts: 91

I think testing emerge metadata or sync, after python-updater and not too long after a previous sync may not be appropriate, because the cache got recently updated, there is a limited amount of files to make portage do the work.

try removing /var/cache/edb/ and run emerge metadata, this really cause all the cache to be rebuilt. In my experience, I am sorry to say it will still stuck at 50 - 52%.