Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge sync takes forever, uses 98% CPU
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5 ... 10, 11, 12  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
rk_cr
n00b
n00b


Joined: 14 Jan 2004
Posts: 12

PostPosted: Tue Oct 04, 2005 5:58 am    Post subject: Reply with quote

I hate to say this but after reading this and testing a sync again it's back...

Damn you KDE. I thought I got rid of you years ago.
Back to top
View user's profile Send private message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 508
Location: North~Sea~Coa~s~~t~~~

PostPosted: Tue Oct 04, 2005 6:21 am    Post subject: Reply with quote

Maybe the problem becomes worse, as there will be more and more KDE split ebuilds, and the KDE monolithic ebuilds will continue until 4.0... :-(

:lol:

Gnome people:
/etc/make.conf
Code:
RSYNC_EXCLUDEFROM="/etc/portage/rsync_excludes"
/etc/portage/rsync_excludes
Code:
- kde-*/
- metadata/cache/kde-*/
...and to prevent emerge --metadata to calculate the cache again with the outdated stuff: remove /usr/portage/kde-*/ and /usr/portage/metadata/cache/kde-*/ manually.

Last edited by lost+found on Thu Oct 20, 2005 8:00 am; edited 3 times in total
Back to top
View user's profile Send private message
Sachankara
l33t
l33t


Joined: 11 Jun 2004
Posts: 696
Location: Stockholm, Sweden

PostPosted: Tue Oct 04, 2005 12:39 pm    Post subject: Reply with quote

Yep, the problem is back. *lol* :/ :(

real 13m31.972s
user 9m54.254s
sys 0m30.140s
_________________
Gentoo Hardened Linux 2.6.21 + svorak (Swedish dvorak)
Back to top
View user's profile Send private message
widremann
Veteran
Veteran


Joined: 14 Mar 2005
Posts: 1314

PostPosted: Tue Oct 04, 2005 1:18 pm    Post subject: Reply with quote

It never went away for me, but I only sync once every two weeks or so. There may have been a period in between there when it went away. It takes upwards of 20 minutes now to complete the whole sync operation.
Back to top
View user's profile Send private message
steves
n00b
n00b


Joined: 10 Jan 2005
Posts: 30

PostPosted: Tue Oct 04, 2005 10:15 pm    Post subject: Reply with quote

I had a similar problem and used the new command eclean to clean up distfiles and packages. It may have been just chance but the next updates ran as normal and no delay through 50% and far shorter time.

You need to use the latest version of gentoolkit though.
Back to top
View user's profile Send private message
Deep-VI
n00b
n00b


Joined: 09 Jan 2005
Posts: 18

PostPosted: Thu Oct 06, 2005 4:01 pm    Post subject: Reply with quote

Ahh, it's much better now as of portage-2.0.51.22-r3. The speed is back!
_________________
Gentoo/GNU Linux: Choice is power - have it YOUR way.
Back to top
View user's profile Send private message
stef
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jul 2003
Posts: 93

PostPosted: Fri Oct 14, 2005 11:16 am    Post subject: Reply with quote

yes, on my k6 300 updating the portage cache takes way > 30 min and stays long time at 50 %
(no binary packages - no /usr/portage/packages dir)
My system has enough free memory, so no swap is used...
some times ago it only took a few minutes ... so performance seems to get worse (or it's just because of bigger portage tree)
Back to top
View user's profile Send private message
toocO.ol
n00b
n00b


Joined: 06 Oct 2005
Posts: 16

PostPosted: Fri Oct 14, 2005 2:56 pm    Post subject: most retarded thing about Gentoo Reply with quote

hi I agree that this is a real problem which devs don't want to face or answer angrily that your are doing something stupid and it's not PORTAGE faults. And it's pretty much Portage/Gentoo fault. Why should any user have to do outside maneouvres in order to fix this issue? No other distro has this problem when I used Debian their package manager was such a breeze even on damn ol' putter. In comparison Portage spends ages SCREECHING the hard drive to do emerge -s foo and emerge --syn and emerge -whatever.

So since emerge and portage are such a unusable piece of crap software the only choice to fix this issue is simply to switch to Eix and Portage cdb. Or to migrate to FreeBSD as apparently many are doing due to this issue.

:-(
Back to top
View user's profile Send private message
kev009
n00b
n00b


Joined: 17 Oct 2005
Posts: 43
Location: Tempe, Arizona

PostPosted: Mon Oct 17, 2005 5:41 am    Post subject: Reply with quote

I too was experiancing the 50% hang up. I cannot say with absolute certainty as I did not perform the tasks atomically, but I sped my horridly slow (40minute!!!) syncs to roughly 3 minutes. Firstly, I wiped /usr/portage and /var/cache/edb. I then reinstalled a portage snapshot and synced but the sync improved only marginally (maybe a minute or two). This tells me a few things, first although fragmentation plays a role in the speed of syncs, it is not the root problem we are experiancing. Next I proceeded to update Python to 2.4.2 and run python-cleaner. My syncs were instantly an order of magnitude faster! I ask anyone experiancing this issue try my method atomically (IE don't do the disk wipe first), and post results to confirm or deny this fix.
_________________
http://www.kev009.com
Back to top
View user's profile Send private message
alinv
Guru
Guru


Joined: 19 Nov 2002
Posts: 395
Location: Bucharest

PostPosted: Mon Oct 17, 2005 7:23 am    Post subject: Reply with quote

Sounds interesting.
What is this python-cleaner? I can't find any reference of it.
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B.
Back to top
View user's profile Send private message
schnake
n00b
n00b


Joined: 03 Dec 2003
Posts: 49
Location: Siegburg / Germany

PostPosted: Mon Oct 17, 2005 8:29 am    Post subject: Reply with quote

Here is what I found searching the net for "python cleaner":

http://www.appealinggifts.com/cord-cordless-python-convertible-vacuum-cleaner.html

LOL
Back to top
View user's profile Send private message
radulucian
Apprentice
Apprentice


Joined: 05 Jan 2004
Posts: 151
Location: Bucharest Romania

PostPosted: Mon Oct 17, 2005 12:26 pm    Post subject: Reply with quote

did you mean python-updater ?
Back to top
View user's profile Send private message
ethrandil
n00b
n00b


Joined: 19 Oct 2003
Posts: 20

PostPosted: Mon Oct 17, 2005 11:30 pm    Post subject: Reply with quote

Hi,
I tried to get better results with an loopback-device and let me say: I got them!

I created a 3G big file with 'dd if=/dev/zero of=portage-file bs=1K count=3M' and formated it with reiserfs.
Then I mounted it ('mount portage-file portage-file-mount -t reiserfs -o loop') copied /usr/portage onto it, and made /usr/portage a symlink to the new filesystem.

I ran 'emerge --metadata' in:
real 2m33.763s
user 0m23.443s
sys 0m7.403s

and had no noticeable hang at 50%.
The speed before was... i did't want to test it again because it was definitly TOO LONG! about > 15 min!

So I guess this could be a fragmentation problem. Maybe I'll bench different Filesystems on the loopfile later.

- Eth
Back to top
View user's profile Send private message
alinv
Guru
Guru


Joined: 19 Nov 2002
Posts: 395
Location: Bucharest

PostPosted: Tue Oct 18, 2005 12:48 am    Post subject: Reply with quote

Using anydbm as a portage backend seems to speed up things. Now, emerge --metadata takes about 3 minutes to complete, w/o any slowdown around 50%.
But I noticed another strange thing: the first emerge -pu world after updating cache takes about 10 minutes, while subsequent runs take less than a minute.
I don't get it :(
_________________
Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.
S.B.
Back to top
View user's profile Send private message
DrZoidberg
Tux's lil' helper
Tux's lil' helper


Joined: 03 May 2003
Posts: 119
Location: New Port Richey, Florida

PostPosted: Tue Oct 18, 2005 1:08 am    Post subject: Reply with quote

I've had this problem for over a year running on PPC. I have had very slow "Updating Portage cache" with the hang at 50% on my 500MHz G4 pismo powerbook. I thought it was just because my laptop was old. Then I had the same problem on my G4 1.2GHz iBook and on a 3GHz pentium 4 desktop!
Back to top
View user's profile Send private message
aethyr
Veteran
Veteran


Joined: 06 Apr 2003
Posts: 1085
Location: NYC

PostPosted: Tue Oct 18, 2005 1:59 am    Post subject: Reply with quote

nevermind, again:
https://forums.gentoo.org/viewtopic-p-2815795.html#2815795


Last edited by aethyr on Thu Oct 20, 2005 10:23 pm; edited 2 times in total
Back to top
View user's profile Send private message
kev009
n00b
n00b


Joined: 17 Oct 2005
Posts: 43
Location: Tempe, Arizona

PostPosted: Tue Oct 18, 2005 4:39 am    Post subject: Reply with quote

radulucian wrote:
did you mean python-updater ?

Yes, sorry folks.. getting perl and python programs confused :-).
_________________
http://www.kev009.com
Back to top
View user's profile Send private message
mrv
Tux's lil' helper
Tux's lil' helper


Joined: 29 Mar 2004
Posts: 114
Location: Oulu, Finland

PostPosted: Tue Oct 18, 2005 4:01 pm    Post subject: Reply with quote

I've also had this issue for 1-2 weeks with a stable system (only couple of ~amd64 packages). I'm running sys-apps/portage-2.0.51.22-r3 and dev-lang/python-2.4.1-r1. The cache updating phase of emerge --sync sits a very long time at 52 %.

-mrv-
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1344
Location: Mississippi USA

PostPosted: Tue Oct 18, 2005 4:48 pm    Post subject: Reply with quote

Well I have three rigs and have noticed the same thing.

Rig one is in the sig and has been running for a while, since about my join date here. I did copy the files, including distfiles, to a seperate partition, delete the old, then copy it back again. /usr/portage is on it's own partition by the way. We'll see if that defrags any and helps. reiserfs all the way around too.

Rig two is a server setup that runs folding and has /boot and / and that is all. Same thing at about 50% though. It is a 800MHz rig with 128MBs of ram.

Rig three is a Compaq Proliant 6000 with quad CPUs and 128MBs of ram set up as a server. This is a very recent install and it was slow from the get go. It hung a bit at 50% after the first sync. It to has /boot and / and that's it.

Rigs two and three runs folding and that is about it. If someone wants me too, I will post the make.conf and/or emerge info if you need it. Just let me know.

Since I am on dial-up, I have noticed that it takes longer to download all the files too. I think when I first started using Gentoo, there was about 60 or 70,000 files, there was about 128,000 last night. Seems like that may have something to do with it but I'm not etching that in stone either. :?

Still LOVE my Gentoo though. I do wish I had DSL though. Of course, I have been dating a lady in Mobile and may move there, they will have DSL there I'm sure. I suspect that about two weeks after I move, they will send me a email that DSL is available here. :roll: They put A/C in our schools the year after I graduated too. 8O That's my luck in life. I'm one in a few thousand with my skin disorder too. If it wasn't for bad luck, I would have no luck at all. My lady is a silver lining though. She is super sweet. :D I need to loose weight and she needs to gain some. She only weighs 97 lbs. 8O I LOVE every ounce though. I have to, there aren't many ounces there. :oops:

Later

:D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
9TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
shimbob
Tux's lil' helper
Tux's lil' helper


Joined: 13 Sep 2003
Posts: 85

PostPosted: Wed Oct 19, 2005 4:42 am    Post subject: Reply with quote

4 machines here showing the same problem. 50 and 51% take minutes to go through, but it speeds along fine before and after those percentages. emerge is hogging 99% cpu

It's a recent thing, within the past 2-3 weeks and came rather sudden.

Reiserv4 on all machines, one 733mhz P3, two 3ghz P4 and one smp amd 2400+ MP machine.
Back to top
View user's profile Send private message
StringCheesian
l33t
l33t


Joined: 21 Oct 2003
Posts: 887

PostPosted: Wed Oct 19, 2005 4:57 am    Post subject: Reply with quote

Same problem on reiserfs 3.6

sys-apps/portage-2.0.53_rc5
net-misc/rsync-2.6.6
sys-kernel/gentoo-sources-2.6.13-r3
In /etc/make.conf: SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
Athlon XP 3200 with 512 MB ram, no other significant processes besides KDE, Konsole, and Firefox
Back to top
View user's profile Send private message
kev009
n00b
n00b


Joined: 17 Oct 2005
Posts: 43
Location: Tempe, Arizona

PostPosted: Wed Oct 19, 2005 5:33 am    Post subject: Reply with quote

Well, the symtoms are now appearing on my Opteron 248 with a U320 10k RPM disk! Although not as drastic as some of my other machines, I guess none are immune.
_________________
http://www.kev009.com
Back to top
View user's profile Send private message
mrv
Tux's lil' helper
Tux's lil' helper


Joined: 29 Mar 2004
Posts: 114
Location: Oulu, Finland

PostPosted: Wed Oct 19, 2005 7:57 am    Post subject: Reply with quote

I don't know if this is a "solution", but I ran /usr/sbin/python-updater and after that "emerge metadata" no longer took so long (didn't stuck at 52 %).

Let's see if the situation is still same tomorrow...

EDIT: Ok, it didn't work. "emerge --sync" still stucks at 52 % and takes the whole CPU time...

-mrv-
Back to top
View user's profile Send private message
marsclic
n00b
n00b


Joined: 02 Mar 2003
Posts: 74

PostPosted: Wed Oct 19, 2005 8:30 am    Post subject: python-updater trick worked for me Reply with quote

I have an athlon64 and a Reiser4 fs on a 4 disk raid-0, and the emerge sync process was taking about 5 minutes, much much longer than it used to be. It also hanged for a long time at the 50% mark, without any disk or network activity. I ran python-updater, and emerge --metadata took very little time to complete. So to confirm, I did a "time emerge sync" again, and this is what I got:
Code:

deleting x11-libs/gtk+/files/digest-gtk+-2.6.7

Number of files: 127953
Number of files transferred: 50
Total file size: 101973845 bytes
Total transferred file size: 156661 bytes
Literal data: 156661 bytes
Matched data: 0 bytes
File list size: 3045801
Total bytes sent: 1149
Total bytes received: 3101344

sent 1149 bytes  received 3101344 bytes  74758.87 bytes/sec
total size is 101973845  speedup is 32.87

>>> Updating Portage cache:  100%

real    1m3.832s
user    0m8.826s
sys     0m4.950s


This now is substantially faster. Many thanks for who suggested this fix. :D
Back to top
View user's profile Send private message
salivian
Tux's lil' helper
Tux's lil' helper


Joined: 15 Sep 2002
Posts: 91

PostPosted: Thu Oct 20, 2005 12:45 am    Post subject: Reply with quote

I think testing emerge metadata or sync, after python-updater and not too long after a previous sync may not be appropriate, because the cache got recently updated, there is a limited amount of files to make portage do the work.

try removing /var/cache/edb/ and run emerge metadata, this really cause all the cache to be rebuilt. In my experience, I am sorry to say it will still stuck at 50 - 52%.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3, 4, 5 ... 10, 11, 12  Next
Page 4 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum