[HOWTO] Speeding up portage with cdb -- UPDATE

cazort · Posted: Fri Mar 24, 2006 3:42 am Post subject:

OK...what I don't understand is why this has been known for well over a year, and yet, it's not integrated into the basic portage yet, whereas developers have engaged (and succesfully, with beautiful results) in much bigger, more complex projects such as the kde split ebuilds.

Why can't this be integrated into the main portage tree? I'm a little bit reluctant to use something like this that is still experimental and "unsupported" on a production server...and yet on this very server it takes FOREVER to emerge sync...really ridiculous. It's taking almost an hour, and using most of the server's resources, on a celeron-2.0ghz machine.

I dunno? It just seems like...this should be integrated into the main portage thing? Portage just keeps getting slower and slower as it grows...and this "solution" remains unsupported and out of the mainstream.

Is there something holding it back?
_________________
Alex Zorach | Teacology | RateTea | Why This Way

pussi · l33t Joined: 08 May 2004 Posts: 727 Location: Finland

cazort · Posted: Wed Mar 29, 2006 4:50 pm Post subject:

Ok...read that part...damn...hmm...

There has to be a way around this? Obviously, we are lightyears ahead of many other OS's, especially a particular one that is not worthy of being named. However...we need to think up a creative solution. Portage keeps growing...the rate at which it is slowing down seems to be exceeding the rate at which the hardware is getting faster. Portage used to sync faster on my old boxes than it does on my newer one now...and like I said, the sync speed is so slow as to make the system unusable.

What other options are there?

Here's a really simple solution (feel free to point out my stupidity if there is a basic reason why it wouldn't work):

Anything that depends on the external module written in C, it could run as a separate process? This is how most UNIX applications work anyway. What would be the problem with that? If it segfaults, then it would not segfault the portage process. You could then write error-handling code to fall back on a safer mechanism in the case that the database part of it broke. It seems like all this would not be terribly difficult to do either.

Am I missing something?
_________________
Alex Zorach | Teacology | RateTea | Why This Way

cazort · Posted: Wed Mar 29, 2006 4:54 pm Post subject:

maybe I just need to play around with 2.1...it seems that maybe they've solved the speed problem some other way?

:-)

_________________
Alex Zorach | Teacology | RateTea | Why This Way

wah · Posted: Wed Mar 29, 2006 6:43 pm Post subject:

Just a "hey, I like it" post:

Pre-cdb:

binaural · Tux's lil' helper Joined: 02 Dec 2004 Posts: 138

I'm with wahman... this really kicks ass, it's pretty much the standard "Thing I do for all my gentoo boxes" now
_________________
Happy vimming!

behd · Apprentice Joined: 11 Feb 2003 Posts: 155

w0000t... well done, great job !

another satisfied customer

(only downside, being a hypothetically risk of segfault once in a while ?!?
with all the time saved by your code, I could probably afford this risk, and
spend a bit time if it ever happen).

Bigun · Advocate Joined: 21 Sep 2003 Posts: 2196

schachti · Posted: Fri May 05, 2006 1:04 pm Post subject:

Hmm, on one system, it works well, but on another system, I always get:

optiluca · Posted: Fri May 05, 2006 4:53 pm Post subject:

schachti · Posted: Fri May 05, 2006 4:57 pm Post subject:

Did you create /usr/lib/portage/pym/portage_db_cdb.py? Do you have portage 2.1 (if yes: http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb)?
_________________
Never argue with an idiot. He brings you down to his level, then beats you with experience.

How-To: Daten verschlüsselt auf DVD speichern.

optiluca · Posted: Fri May 05, 2006 5:18 pm Post subject:

portage-2.1_pre10-r2...
And it ISN'T fast...

BitJam · Posted: Sat May 06, 2006 4:46 am Post subject:

I get the same error as optiluca with portage-2.1_pre10-r3.

Phlogiston · Posted: Mon May 08, 2006 1:07 pm Post subject:

Eix does not work when I set cdb in eixrc. Is there a known workaround for this? It just does not show any result from the portage directory... only from the overlays, which are not cached.
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor

appro · Posted: Thu May 11, 2006 11:55 am Post subject:

https://bugs.gentoo.org/show_bug.cgi?id=132915
_________________
If a packet hits a pocket on a socket on a port, and the bus is interrupted and the interrupt's not caught, then the socket packet pocket has an error to report.

Phlogiston · Posted: Fri May 12, 2006 11:56 am Post subject:

langec · Posted: Sun May 14, 2006 4:41 pm Post subject: Thanks!

Thanks a lot for that speedup!

Autie · Posted: Sat Jun 10, 2006 9:08 pm Post subject:

Changed as the bug said:

theoakwise · n00b Joined: 15 May 2004 Posts: 37

Bigun · Advocate Joined: 21 Sep 2003 Posts: 2196

Guh.... back to slow portage again... anyway this could be re-written?
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

monkey prince · n00b Joined: 12 Jun 2006 Posts: 2

check out the wiki for cdb and portage:

http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb

i had the same error messages until i changed my /etc/portage/modules to say

Bigun · Advocate Joined: 21 Sep 2003 Posts: 2196

Doesn't seem to be helping:

Bigun · Advocate Joined: 21 Sep 2003 Posts: 2196

Nevermind:

Somehow I this file had gotten deleted or just wasn't created somehow... I dunno, I got it from the wiki link above:

/usr/lib/portage/pym/cache/cdb.py
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

chl · n00b Joined: 13 May 2003 Posts: 72 Location: Munich

I use the cdb-caching method for a long time.

But since portage 2.1.1 became stable, when syncing or updating the cache with "emerge --metadata" the "Updating Portage cache: " part has become significally slower.

Under portage 2.1 the update of the cache from 0% to 100% took just a few seconds, even if I hadn't synced for a long time. But with portage 2.1.1 it takes up to five minutes.

Even if I run "emerge --metadata" several times, the second and further runs take as long as the first one. This happens also, if I disable the cdb-caching.

This is perfectly reproducible. If I downgrade to portage 2.1-r2 it runs like hell, but with the actual stable version (2.1.1-r2) it's slow as a snail.

What are the reasons and is there any possibility to have the old speed also with portage 2.1.1?

CHL
_________________
The difficulty in designing something completely foolproof, is not to underestimate the ingenuity of complete fools.

steveL · Posted: Wed Dec 20, 2006 5:52 am Post subject:

I haven't used the cdb-caching method, although this topic made me want to- until I read the last page! I agree about the speed of portage cache updates; it's definitely got a lot slower.