Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge sync takes forever, uses 98% CPU
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3 ... 10, 11, 12  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Fri Sep 23, 2005 10:23 pm    Post subject: Emerge sync takes forever, uses 98% CPU Reply with quote

Not sure what happened all the sudden but running emerge --sync uses up 98% CPU and takes forever. It used to take a couple minute to update - now it's like 20-30 mins... It's slow on the "Updating Portage Cache... .X%" - The rsync is fine...

Any ideas? I looked through the forums and didn't see anything like this... it just started this last week...

I sync about every day or two and am updated on all packages.... can't figure out what's doing this.


Last edited by xtaski on Sat Sep 24, 2005 10:25 pm; edited 2 times in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43755
Location: 56N 3W

PostPosted: Fri Sep 23, 2005 10:34 pm    Post subject: Reply with quote

xtaski,

Do you have a lot of binary packages?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Fri Sep 23, 2005 10:42 pm    Post subject: Reply with quote

nope; only binary is probably openoffice and nvidia... everything else is built from source...
Back to top
View user's profile Send private message
ldoe3000
Tux's lil' helper
Tux's lil' helper


Joined: 19 Jul 2005
Posts: 78
Location: Portland, OR

PostPosted: Sat Sep 24, 2005 1:18 am    Post subject: Updating Portage cache taking a long time lately Reply with quote

Anyone else experiencing really long "Updating Portage cache:" times when doing an emerge sync? Even if I do an "emerge metadata", it still takes about 3-4 minutes on that step alone. In the past it took under 1 minute.. Is it just me?
_________________
Athlon64 3500+ MSI Neo2 Platinum
2X74 WD Raptor Raid 0
2X512MB Crucial Ballistix
ATI x800XT PE (8.18.6)
SB Audigy 2 ZS
Dell 24" Widescreen LCD
Dual boot: Gentoo x86_64 2.6.13-r4 , Windows XP x64
Back to top
View user's profile Send private message
cantao
Apprentice
Apprentice


Joined: 07 Jan 2004
Posts: 166

PostPosted: Sat Sep 24, 2005 1:26 am    Post subject: Same here... Reply with quote

Yep, same behavior here... I thought it was just my imagination :)

Any clues?

Cheers, Cantão!
Back to top
View user's profile Send private message
ldoe3000
Tux's lil' helper
Tux's lil' helper


Joined: 19 Jul 2005
Posts: 78
Location: Portland, OR

PostPosted: Sat Sep 24, 2005 2:12 am    Post subject: Reply with quote

None, except we're up to 126000 files.. Perhaps Portage has hit some threshold where it can't really keep up with the volume now.. Total guess there.
_________________
Athlon64 3500+ MSI Neo2 Platinum
2X74 WD Raptor Raid 0
2X512MB Crucial Ballistix
ATI x800XT PE (8.18.6)
SB Audigy 2 ZS
Dell 24" Widescreen LCD
Dual boot: Gentoo x86_64 2.6.13-r4 , Windows XP x64
Back to top
View user's profile Send private message
satanskin
Guru
Guru


Joined: 25 Apr 2005
Posts: 353

PostPosted: Sat Sep 24, 2005 3:02 am    Post subject: Reply with quote

They should switch it over and use some sort of database with it. Maybe support one of the C++ versions of portage...

Regardless, I'm having the same thing. Takes a while.
Back to top
View user's profile Send private message
barum87
n00b
n00b


Joined: 28 Mar 2005
Posts: 62

PostPosted: Sat Sep 24, 2005 3:18 am    Post subject: Reply with quote

Me too.
I just thought that my system wasn't optimized enough, but I guess it's not just my problem.
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Sat Sep 24, 2005 7:52 am    Post subject: Reply with quote

In deed, portage should become faster.
_________________
http://tam.belchenstuermer.de/
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3707
Location: Hamburg

PostPosted: Sat Sep 24, 2005 8:00 am    Post subject: Reply with quote

An "emerge sync" is since some days/weeks significantly slower than before. I see "Updating Portage cache: 50%" for 1-2 minute(s) or longer before it changes.
This is independent from the general question how the process can be speed up.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43755
Location: 56N 3W

PostPosted: Sat Sep 24, 2005 8:26 am    Post subject: Reply with quote

xtaski,

Sorry, I wasn't clear.

Do you have
Code:
FEATURES="buildpkg"
set in /etc/make.conf, even by defualt ?
You need not have installed any binary packages. Look in
Code:
emerge --info
since that will show you things which are on by defualt too.

This makes portage save a binary of everything you build, at the end of emerge sync it does a lot checking of these binaries.
They are saved in /usr/portage/packages by defualt.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Gentii
Guru
Guru


Joined: 01 Feb 2004
Posts: 306

PostPosted: Sat Sep 24, 2005 11:30 am    Post subject: Reply with quote

Yep, I have exactly the same problem. In fact I just asked on gentoo chan why this step was getting so slow.. Anyone told me to give cdb module a try ( https://forums.gentoo.org/viewtopic-t-261580-highlight-cdb+module.html ).
So what's the best workaround (easiest and cleanest to set up) ? cdb , one c++ portage like sportage, or something else?

Btw, this post has nothing to do with amd64, maybe it should be moved :)
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3707
Location: Hamburg

PostPosted: Sat Sep 24, 2005 11:55 am    Post subject: Reply with quote

I have a similar problem, after 50% is completed emerge uses full cpu power but that's all for a while https://forums.gentoo.org/viewtopic.php?p=2749780#2749780
BTW here is my config
Code:

CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"

USE="acpi aim apache2 browserplugin cardbus cddb cdparanoia dvd dvdr dvdread
     ethereal font-server innodb jai javascript jimi live mbox mmx
     mmxext mng nagios-dns nagios-ntp nagios-ping nagios-s network
     no-old-linux nsplugin pcmcia real slp smime sse sse2 subversion svg
     usb userlocales webdav wifi xanim xfs -apm -gnome"
LINGUAS="de en"

MAKEOPTS="-j2"
PORTAGE_NICENESS="3"
PORT_LOGDIR="/var/log/portage"
FEATURES="ccache"
CCACHE_SIZE="1G"


Last edited by toralf on Sat Sep 24, 2005 6:09 pm; edited 1 time in total
Back to top
View user's profile Send private message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Sat Sep 24, 2005 5:33 pm    Post subject: Reply with quote

Quote:
Do you have
Code:
FEATURES="buildpkg"
set in /etc/make.conf, even by defualt ?
You need not have installed any binary packages. Look in
Code:
emerge --info
since that will show you things which are on by defualt too.

This makes portage save a binary of everything you build, at the end of emerge sync it does a lot checking of these binaries.
They are saved in /usr/portage/packages by defualt.


Ok, yes I do have "buildpkg" - how do I "fix" this then?
Back to top
View user's profile Send private message
kmare
l33t
l33t


Joined: 20 Nov 2004
Posts: 619
Location: Thessaloniki, Greece

PostPosted: Sat Sep 24, 2005 5:48 pm    Post subject: Reply with quote

same problem here..
Back to top
View user's profile Send private message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Sat Sep 24, 2005 6:19 pm    Post subject: Reply with quote

Not sure if this helps, but here's the summary after a --sync:

Code:
Number of files: 125563
Number of files transferred: 1198
Total file size: 99513410 bytes
Total transferred file size: 2450380 bytes
Literal data: 2450380 bytes
Matched data: 0 bytes
File list size: 2862533
File list generation time: 15.113 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 33240
Total bytes received: 5376530

sent 33240 bytes  received 5376530 bytes  53298.23 bytes/sec
total size is 99513410  speedup is 18.40

>>> Updating Portage cache:  100%
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43755
Location: 56N 3W

PostPosted: Sat Sep 24, 2005 6:50 pm    Post subject: Reply with quote

xtaski,

If you never use the binary packages, you can turn off FEATURES="buildpkg" and get rid of the /usr/portage/packages directory.
emerge sync will not to the binary package bit at the end.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
widremann
Veteran
Veteran


Joined: 14 Mar 2005
Posts: 1314

PostPosted: Sat Sep 24, 2005 8:33 pm    Post subject: Reply with quote

I have the same problem and I don't have "buildpkg" set. It just takes forever and the CPU has high utilization. I don't even have a /usr/portage/packages directory, either.
Back to top
View user's profile Send private message
Nijinski
n00b
n00b


Joined: 28 Nov 2004
Posts: 72
Location: Fife, Scotland

PostPosted: Sat Sep 24, 2005 8:59 pm    Post subject: Reply with quote

Same problem here, not limited to AMD64 either, getting it on SPARC and x86 as well.

Big pause at 50% then starts off again.
_________________
Help ! Help ! I'm being repressed
Back to top
View user's profile Send private message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Sat Sep 24, 2005 9:53 pm    Post subject: Reply with quote

Thanks Neddy! That fixed my problem - guess I never really knew why I kept including that feature... now I know not to.
Back to top
View user's profile Send private message
cmtaylor
n00b
n00b


Joined: 09 Aug 2004
Posts: 36
Location: Houghton, MI

PostPosted: Sat Sep 24, 2005 10:24 pm    Post subject: slow portage Reply with quote

i'm having the same problem as wildremann -- i don't have buildpkg set, and i don't have the /use/portage/packages directory. any other possibilities?
Back to top
View user's profile Send private message
xtaski
Apprentice
Apprentice


Joined: 20 Dec 2004
Posts: 168
Location: New York, NY

PostPosted: Sat Sep 24, 2005 10:25 pm    Post subject: Reply with quote

Can you post your make.conf? Maybe there's something else in there...
Back to top
View user's profile Send private message
The Mountain Man
l33t
l33t


Joined: 03 Sep 2003
Posts: 643

PostPosted: Sun Sep 25, 2005 2:58 am    Post subject: Reply with quote

Glad I'm not alone. Takes some minutes and uses almost 100% of my CPU. Wonder what's up?
_________________
I suck at signatures.
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10306
Location: Córdoba (Spain)

PostPosted: Sun Sep 25, 2005 3:11 pm    Post subject: [Slow Portage] - Retrieving metadata - (Open) Reply with quote

The tittle says it all: my box can spend 10-20 minutes (I havent meassured exactly, but is noticeably slower then before) to retrieve all the metadata, after a sync. I have experienced this problem since a couple of months or so. The filesystem is ext2, that is fast, and I checked that is 100% error free. The cpu is mostly idle when doing that, and there is enough free memory and no swap at all when performing this operation.

I noticed that the number of files in portage has grown in the last months a lot, but I dont know if that is what is causing this odd thing. I use a couple of overlays. One in /usr/local/portage with 10 or 15 ebuilds only in it. The other is bigger, in /usr/local/gentopia, with the ebuild from here: http://gentopia.gentooexperimental.org./wiki

Has anyone experienced this slowdown in the last months, or is it just me? It worries my, since, if the portage size is the real problem this could become a serious problem in the next few months... :(
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 508
Location: North~Sea~Coa~s~~t~~~

PostPosted: Sun Sep 25, 2005 4:03 pm    Post subject: Reply with quote

I think it's a filesystem problem: inefficient storage of small files, spread over the hd everywhere...
Some time ago, i put back a Portage snapshot from a tar.bz2, after deleting /usr/portage... after that it was fast again... Maybe ReiserFS would do it better (it's fast on small files)? It should be used for /usr/portage and /var/cache/edb then. I'm using JFS, so i decided to cut down the Portage tree with rsync_excludes, very fast, but you will have to do some manual work when emerging new packages.


Last edited by lost+found on Mon Sep 26, 2005 5:50 am; edited 1 time in total
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2, 3 ... 10, 11, 12  Next
Page 1 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum