Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[HOWTO] Speeding up portage with cdb -- UPDATE
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
lavacano
Apprentice
Apprentice


Joined: 29 May 2005
Posts: 190
Location: Poulsbo, WA

PostPosted: Tue Aug 02, 2005 4:38 pm    Post subject: Reply with quote

yeah this rocks the box. I saw a dev on the bug ticket take real interest, maybe there is a possibility for this to be default?
_________________
Sincerely,

Chadwick Ferguson
Back to top
View user's profile Send private message
gwolf
Guru
Guru


Joined: 05 May 2005
Posts: 374
Location: Estonia

PostPosted: Wed Aug 03, 2005 8:35 am    Post subject: Reply with quote

Really good job!

Note that this script also speeds up equery (a part of gentoolkit).
_________________
War is peace. Freedom is slavery. Ketchup is a vegetable.
Back to top
View user's profile Send private message
burn-gentoo
n00b
n00b


Joined: 17 Jun 2004
Posts: 13

PostPosted: Tue Sep 06, 2005 4:11 pm    Post subject: Reply with quote

I decided to do some benchmarks on this handy mod, on multiple systems to get an idea of how well it actually speeds up the portage cache. This is what I've come up with so far. Please keep in mind that these systems are hugely different in terms of RAM and disk type/configuration.

Test Conditions on all systems
Code:
start with 0.00 load
15 minute lapse between each command for memory clearing
same timestamp for portage (equal files)


System specs are as follows:
TEST1 OS: Linux 2.6.11-hardened-r15/i686 - Distro: Gentoo 1.6.13 - CPU: 2 x Intel Pentium 4 (2992.845 MHz) - Processes: 111 - Memory Usage: 385.70MB/3026.82MB (12.74%) - NOTE: is using md raid 1 (disk writes will be pokey, disk reads will be quite fast)

TEST2 OS: Linux 2.6.12-gentoo-r6/i686 - Distro: Gentoo 1.6.13 - CPU: 2 x Intel Pentium 4 (2599.028 MHz) - Processes: 63 - Memory Usage: 198.92MB/1010.88MB (19.68%)

TEST3 OS: Linux 2.6.12-gentoo-r9/i686 - Distro: Gentoo 1.6.13 - CPU: Pentium III (Coppermine) (737.194 MHz) - Processes: 62 - Memory Usage: 58.54MB/122.71MB (47.70%)


Without CDB - regular ole portage
Code:
TEST1
# time emerge metadata
real    0m50.483s
# time emerge -s python
real    0m1.457s

TEST2
# time emerge metadata
real    0m15.253s
# time emerge -s python
real    0m1.499s

TEST3
# time emerge metadata
real    9m17.262s
# time emerge -s python
real    0m49.744s


With CDB portage mod - (keep in mind that the emerge metadata test is done with an UPDATE to the portage cache. The CDB files have already been built, timing the initial build process is pointless)
Code:
TEST1
# time emerge metadata
real    0m13.186s
# time emerge -s python
real    0m1.368s

TEST2
# time emerge metadata
real    0m13.923s
# time emerge -s python
real    0m1.325s

TEST3
 # time emerge metadata
real    1m47.600s
# time emerge -s python
real    0m13.332s


Looks to me like it makes very little difference on newer systems, but damn, what a difference on the mothball in there.
Back to top
View user's profile Send private message
gwolf
Guru
Guru


Joined: 05 May 2005
Posts: 374
Location: Estonia

PostPosted: Wed Sep 07, 2005 2:08 pm    Post subject: Reply with quote

Just wondering when will it be compatible with eix?
_________________
War is peace. Freedom is slavery. Ketchup is a vegetable.
Back to top
View user's profile Send private message
appro
Tux's lil' helper
Tux's lil' helper


Joined: 21 Jul 2003
Posts: 144
Location: Germany/Rosenheim

PostPosted: Sat Sep 10, 2005 5:53 pm    Post subject: Reply with quote

eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code:
PORTDIR_CACHE_METHOD="cdb"

into your /etc/eixrc.
_________________
If a packet hits a pocket on a socket on a port, and the bus is interrupted and the interrupt's not caught, then the socket packet pocket has an error to report.
Back to top
View user's profile Send private message
gwolf
Guru
Guru


Joined: 05 May 2005
Posts: 374
Location: Estonia

PostPosted: Sat Sep 10, 2005 7:23 pm    Post subject: Reply with quote

appro wrote:
eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code:
PORTDIR_CACHE_METHOD="cdb"

into your /etc/eixrc.


And it really was that simple! :)
Big thanks!
_________________
War is peace. Freedom is slavery. Ketchup is a vegetable.
Back to top
View user's profile Send private message
whitesouls
Guru
Guru


Joined: 19 Nov 2004
Posts: 358
Location: In Front of My Laptop

PostPosted: Sat Sep 10, 2005 8:06 pm    Post subject: Reply with quote

appro wrote:
eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code:
PORTDIR_CACHE_METHOD="cdb"

into your /etc/eixrc.


Thanks appro, it worked for me well too.. :D thank you tobidope for the CDB.
_________________
whitesouls

Please insert the [SOLVED] tag if your problem is solved in your respective thread.
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sun Sep 11, 2005 7:27 am    Post subject: Reply with quote

it still makes a difference for me at least.

Code:
Family: 15 Model: 4 Stepping: 1 Type: 0 Brand: 0
CPU Model: Pentium 4 (Prescott) [E0]
model name   : Mobile Intel(R) Pentium(R) 4 CPU 3.06GHz
Feature flags:
 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflsh ds acpi mmx fxsr sse sse2 ss ht tm pbe sse3 monitor ds-cpl est tm2 cntx-id xTPR


HD is a slow as hell 4200RPM 80GiB laptop drive.

default
Code:
#  rm -r /var/cache/edb
#  time emerge metadata
real   1m57.441s
user   1m17.038s
sys   0m10.791s


cdb
Code:
#  rm -r /var/cache/edb
#  time emerge metadata
real   0m37.501s
user   0m18.957s
sys   0m2.869s

_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
hongqn
n00b
n00b


Joined: 04 Apr 2005
Posts: 22
Location: Beijing, China

PostPosted: Tue Sep 13, 2005 8:01 am    Post subject: Reply with quote

It looks nice :D
However, I wander what else will this tip benefit me except the speeding up of portage search? I always use eix to search packages and am satisfied with its perfermance.
_________________
God's in his heaven, all's right with the world.
Registered Linux User #396996
Back to top
View user's profile Send private message
bob_111
Apprentice
Apprentice


Joined: 12 Oct 2004
Posts: 155

PostPosted: Tue Sep 13, 2005 12:17 pm    Post subject: Reply with quote

WOW! I just installed this and i am amazed! My PC is rellativly fast :wink: and now i can search almost instantly! Nice work! I don't see why this shouldn't be added to regular portage!

- bob_111
Back to top
View user's profile Send private message
crs
Apprentice
Apprentice


Joined: 19 Apr 2003
Posts: 188

PostPosted: Wed Sep 14, 2005 9:30 am    Post subject: Reply with quote

Hello. I have problem with newest portage version (Portage 1.589-cvs). Its from ebuild: 2.1.0_alpha20050718.
If i have turn on cdb module i get errors:
Code:
# emerge metadata
Traceback (most recent call last):
  File "/usr/bin/emerge", line 10, in ?
    import portage
  File "/usr/lib/portage/pym/portage.py", line 4997, in ?
    portdb=portdbapi(settings["PORTDIR"])
  File "/usr/lib/portage/pym/portage.py", line 2729, in __init__
    gid=portage_gid)
TypeError: __init__() got an unexpected keyword argument 'basepath'

And can't do anything with portage because all the time get this error.
cdb module don't support 2.1.* portage version?
Back to top
View user's profile Send private message
Gentii
Guru
Guru


Joined: 01 Feb 2004
Posts: 306

PostPosted: Sat Sep 24, 2005 11:41 am    Post subject: Reply with quote

Well, I've troubles with sys-apps/portage-2.0.52-r1 as well.

Code:
# emerge metadata
Traceback (most recent call last):
  File "/usr/bin/emerge", line 10, in ?
    import portage
  File "/usr/lib/portage/pym/portage.py", line 7149, in ?
    portdb=portdbapi(settings["PORTDIR"])
  File "/usr/lib/portage/pym/portage.py", line 5134, in __init__
    self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
  File "/usr/lib/portage/pym/portage.py", line 4977, in __init__
    self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
  File "/usr/lib/portage/pym/portage.py", line 1226, in load_best_module
    return load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
    mod = __import__(modname)
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 16, in ?
    USA
NameError: name 'USA' is not defined
Back to top
View user's profile Send private message
sn4ip3r
Guru
Guru


Joined: 14 Dec 2002
Posts: 325
Location: Tallinn, Estonia

PostPosted: Sat Sep 24, 2005 12:09 pm    Post subject: Reply with quote

Gentii wrote:
Well, I've troubles with sys-apps/portage-2.0.52-r1 as well.
...


Already solved.
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Mon Oct 17, 2005 10:08 pm    Post subject: Reply with quote

Oouustanding !

sync no longer takes 3minutes to do the 52nd percentile and searches are so much quicker it's unnerving.

A great contribution. Many thanks. :D
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
wangxiaohu
Apprentice
Apprentice


Joined: 15 Nov 2003
Posts: 248

PostPosted: Tue Oct 18, 2005 9:49 pm    Post subject: Reply with quote

Following the intructions, I switched portage to cdb based. Now it is possible to use SQLite as a backend?
Back to top
View user's profile Send private message
sn4ip3r
Guru
Guru


Joined: 14 Dec 2002
Posts: 325
Location: Tallinn, Estonia

PostPosted: Tue Oct 18, 2005 10:51 pm    Post subject: Reply with quote

wangxiaohu wrote:
Following the intructions, I switched portage to cdb based. Now it is possible to use SQLite as a backend?


It sure is, if you write the code.
Back to top
View user's profile Send private message
wyv3rn
Apprentice
Apprentice


Joined: 18 Aug 2005
Posts: 154
Location: USA

PostPosted: Sat Oct 22, 2005 7:44 pm    Post subject: Reply with quote

Anyone benchmark this against anydbm?
Back to top
View user's profile Send private message
wyv3rn
Apprentice
Apprentice


Joined: 18 Aug 2005
Posts: 154
Location: USA

PostPosted: Sat Oct 22, 2005 9:59 pm    Post subject: Reply with quote

Machine:
Intel P3 Celeron 800Mhz/100Mhz FSB
256MB PC100 CAS 2 RAM
Adaptec 39160 U160 SCSI Controller
Fujitsu MAN3184 10,000RPM SCSI HD

Test methadology:
cd /var/cache/edb/dep/usr/portage ; rm -rf *
change module to be tested (cpickle, anydbm, cdb) in /etc/portage/modules
reboot, then run the following tests:

TEST1:
time emerge metadata
time emerge metadata
TEST2:
time emerge -upDN1 world
TEST3:
time emerge --searchdesc mozilla
time emerge search python

Results:

--- CPICKLE ---
TEST 1:
real 12m20.934s
user 7m52.709s
sys 0m50.588s

real 1m48.220s
user 0m46.159s
sys 0m10.355s

TEST 2:
real 0m18.804s
user 0m11.656s
sys 0m1.270s

TEST 3:
real 1m47.575s
user 0m56.247s
sys 0m10.196s

real 0m8.594s
user 0m3.647s
sys 0m0.617s
----------------------

--- ANYDBM ---
TEST 1:
real 8m34.878s
user 1m9.006s
sys 0m19.395s

real 2m18.461s
user 0m54.045s
sys 0m7.589s

TEST 2:
real 0m25.021s
user 0m13.158s
sys 0m1.146s

TEST 3:
real 2m0.898s
user 1m4.922s
sys 0m7.351s

real 0m15.963s
user 0m3.537s
sys 0m0.755s
--------------------

--- CDB ---
TEST 1:
real 3m9.459s
user 0m54.717s
sys 0m11.465s

real 1m1.524s
user 0m41.711s
sys 0m3.901s

TEST 2:
real 0m14.951s
user 0m10.903s
sys 0m0.718s

TEST 3:
real 1m8.174s
user 0m52.572s
sys 0m3.704s

real 0m4.272s
user 0m3.223s
sys 0m0.381s
--------------------
Back to top
View user's profile Send private message
smithjd15
Tux's lil' helper
Tux's lil' helper


Joined: 12 Mar 2003
Posts: 130
Location: Edmonton, AB, CA

PostPosted: Sun Oct 23, 2005 5:19 pm    Post subject: Reply with quote

I had issues with the version of this script on the initial post. The one on gentoo-wiki worked great. Running them both through Kompare showed extra whitespace on the copy-and-pasted version from this thread that was not there in the wiki version.

If you are having issues with the version in this thread, use the version at the wiki. http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb .

Perhaps the version in this thread can be fixed from the wiki version?
Back to top
View user's profile Send private message
gordin
Guru
Guru


Joined: 11 Oct 2002
Posts: 300
Location: Germany/WI

PostPosted: Tue Oct 25, 2005 11:35 pm    Post subject: Reply with quote

works great on my iBook G3...
updating portage cache took about 45min before... that was really annoying

Now it is taking about 5min... thats a time I can live with.
Back to top
View user's profile Send private message
matobsk
n00b
n00b


Joined: 16 Oct 2005
Posts: 64

PostPosted: Wed Oct 26, 2005 5:10 am    Post subject: Reply with quote

Thank you for finally alleviating the major slowdown at the 50-52% of regenerating my cache.

No matter what I did, regular portage would speed right along to 50%, then use 100% cpu for 8-10 minutes to get to 53%, then fly along again.

This method eliminated that bottleneck, and made the whole portage system faster too.
Back to top
View user's profile Send private message
R. Daneel Olivaw
n00b
n00b


Joined: 06 Jul 2004
Posts: 47

PostPosted: Wed Oct 26, 2005 7:53 pm    Post subject: Re: /me - dumb Reply with quote

slivovice wrote:
thanks sn4ip3r nano cut my line off. what a dumb negligence of me :oops:


IMHO that's more a problem of nano default behaviour which is dumb...
Back to top
View user's profile Send private message
fourteen20
n00b
n00b


Joined: 15 Aug 2003
Posts: 36
Location: Cornwall,uk

PostPosted: Thu Oct 27, 2005 1:52 am    Post subject: Reply with quote

Thanks to all for this great portage modification, emerge sync was starting to annoy me with the time it was taking :D
_________________
I Wanna be a Guru
Back to top
View user's profile Send private message
dirtbag
Guru
Guru


Joined: 18 Feb 2003
Posts: 508
Location: NC

PostPosted: Mon Oct 31, 2005 11:34 pm    Post subject: grrrrr Reply with quote

:evil:
ive been running this on my system and I just updated portage and did
emerge update system and I believe it updated python..
now when I try emerges, I get

dellbox jason # emerge portage
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_cdb.py", line 21, in ?
import cdb
ImportError: No module named cdb
dellbox jason #

any ideas?
Back to top
View user's profile Send private message
pussi
l33t
l33t


Joined: 08 May 2004
Posts: 727
Location: Finland

PostPosted: Tue Nov 01, 2005 12:21 am    Post subject: Reply with quote

Run this as root:
Code:
mv /etc/portage/modules /etc/portage/__modules
python-updater
mv /etc/portage/__modules /etc/portage/modules
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 5 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum