[HOWTO] Speeding up portage with cdb -- UPDATE

PrakashP · Posted: Sun Dec 05, 2004 11:48 am Post subject:

SQLite - see portage-c... (needs less than 1 (sometimes to about 2) seconds to find something for me, cdb needs about 4-5 seconds) But it could also be python slowing down things a lot...

SuperJudge · Posted: Mon Dec 06, 2004 3:49 pm Post subject:

emerge metadata is taking a long time.
_________________
---Powered by Honda

s4kk3 · Posted: Mon Dec 06, 2004 3:53 pm Post subject:

yes. emerge metadata takes long but everything else is 100x faster for me.
_________________
My own filemanager project

steveb · Advocate Joined: 18 Sep 2002 Posts: 4564

SuperJudge · Posted: Mon Dec 06, 2004 4:44 pm Post subject:

tobidope · n00b Joined: 17 Aug 2003 Posts: 24 Location: Germany

SuperJudge · Posted: Mon Dec 06, 2004 5:49 pm Post subject:

It's no problem, I just didn't realize it was supposed to take a long time.

I'll background it.
_________________
---Powered by Honda

tobidope · n00b Joined: 17 Aug 2003 Posts: 24 Location: Germany

I didn't want to be offensive in anyway, but I will also try anydbm. And maybe I'll have to climb down in portage.py for enhancing the cache generation. I see no need for syncing after each key insertion (that's what makes emerge metatdata so slow), but maybe I should ask somebody who knows about that. Somebody here with expert portage knowledge? I think I should go to the "Portage & Pogramming" forum.

TrueDFX · Retired Dev Joined: 02 Jun 2004 Posts: 1348

Wow, I'm impressed. I read that emerge --metadata would be slow, but this...

emerge --metadata in one terminal. emerge -1 glibc in another. emerge -1 glibc is moving the files to / right now, while emerge --metadata is at 82%.

(I'm sure it would have been a *lot* faster if I waited with the glibc installation until --metadata finished.)

As long as searching with emerge is fast afterwards, I'm happy. And it is

mirko_3 · l33t Joined: 02 Nov 2003 Posts: 605 Location: Birreria

If searching is all it improves, I don't see why you don't just use (like I do) esearch: it's in portage, I rebuild the database nightly, and search is not fast - it's instant...
_________________
Non fa male! Non fa male!

Hydraulix · Guru Joined: 12 Dec 2003 Posts: 447

Has this been updated in the new version of Portage?
_________________
It is the fate of operating systems to become free.
- Neal Stephenson

TazG · Guru Joined: 22 May 2004 Posts: 320 Location: Canada

You know I don't think it's as fast as you are saying it is. You see if you emerge -S mozilla twice, it will be faster the second time whether you are using cdb or not because it is cached in memory.
emerge -S mozilla without cdb:

PrakashP · Posted: Wed Dec 08, 2004 2:03 pm Post subject:

But the difference is, cdb is fast from the first time, and even faster the second time when data files are cached...

TazG · Guru Joined: 22 May 2004 Posts: 320 Location: Canada

Actually a fourth emerge -S mozilla (second time with cdb) is only 4 seconds faster than the third.

tobidope · n00b Joined: 17 Aug 2003 Posts: 24 Location: Germany

And the second difference, if you have an older machine with not so much RAM and disk cache most oft the time when you do something portage related the process is waiting for I/O. Of course throwing more RAM and processorpower at something solves almost any performance problem. That's one point of view and mostly it's cheaper to buy more hardware instead of solving a problem efficient. But in my opinion the db backend of portage is something where you could think 5 minutes to enhance it. In an ideal world there should be no need for something like esearch, because the portage cache would be fast enough.

killercow

isn;t it theoretically possible to connect to any mysql server?
If this mysql server sits in the lan on my personal house server all of my other machines should be able to connect to it and search the portage tree on that machine?

How would we do this? im no python whiz (alltough i can do some stuff) so is there a quick way to do this?

Would it also suffice to just update the portage tree on the server (or does portage keep a different tree based on the emerged packages per system?)

tobidope · n00b Joined: 17 Aug 2003 Posts: 24 Location: Germany

@killercow

Read the first post of this thread. There is a reference to the mysql solution.

killercow · Posted: Wed Dec 15, 2004 10:03 am Post subject: noticed the mysql part

could it be possible to keep one portage tree in mysql on a seperate machine? And still use all different clients (my machine, my laptops,my servers, my brothers pc, etc) to use this one portage tree and still have a different set of installed programs?

Mindphaser · Posted: Wed Dec 15, 2004 10:12 pm Post subject:

Very nice, but "emerge sync" is hell slow now....
Is it normal, that before syncing the Portage Cache, he simply deletes it ?
So "Updating Portage Cache" takes as long like "emerge metadata" does....

And, sadly, i dont get a massive speed-up like some other people here.
"emerge kde --search > bla" took 4,5sec. , with cdb-powered-portage, 4,3sec

However, thats better than nothing, but that long "Update Portage Cache" after every sync are realy disturbing....

btw: I have a Athlon XP 2600+ with 512MB RAM

LucaSpiller · Posted: Wed Dec 15, 2004 10:56 pm Post subject:

It seems ok, there doesn't really seem to be much of a speed increase for me - mainly because I don't search that much. Searching is quite a bit quicker but is still rather slow.

Portage-c looks good so I might give that a try.
_________________
:: Luca :: Mac Fag :: Original Macbook, 2g RAM :: Closet Linux user (seasoned with salt and pepper) :: C2D E4400 @ 2ghz, 4g RAM (only 3.2g detected under 64bit...), Nvidia 9600GSO ::

gentoo_lan · Posted: Thu Dec 16, 2004 12:04 am Post subject:

It looks pretty good to me so far. Good work.

thoffmeyer · Posted: Thu Dec 16, 2004 3:35 pm Post subject:

works great over here, im too lazy to post but thanks.

tobidope · n00b Joined: 17 Aug 2003 Posts: 24 Location: Germany

PrakashP · Posted: Sun Dec 19, 2004 11:17 am Post subject:

[Edit] Forget it, I somehow missed the first line...

Yup, it's a *lot* faster on updating database. Very nice.

matroskin · Apprentice Joined: 21 Jan 2003 Posts: 214

after upgrading to new /usr/lib/portage/pym/portage_db_cdb.py
I get this error.