Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[HOWTO] Speeding up portage with cdb -- UPDATE
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
dirtbag
Guru
Guru


Joined: 18 Feb 2003
Posts: 508
Location: NC

PostPosted: Tue Nov 01, 2005 1:13 am    Post subject: that did it! Reply with quote

woohoo! saved by pussi! thanks..

-DB
Back to top
View user's profile Send private message
itsr0y
Tux's lil' helper
Tux's lil' helper


Joined: 22 Dec 2002
Posts: 81

PostPosted: Tue Nov 01, 2005 3:42 am    Post subject: Reply with quote

This is the best Gentoo tip I've seen. That 52% slowdown when doing an 'emerge sync' has been eliminated, plus searching is no longer unusably slow! Thank you!
Back to top
View user's profile Send private message
bravecobra
Tux's lil' helper
Tux's lil' helper


Joined: 26 Dec 2002
Posts: 130
Location: Planet Earth (sometimes)

PostPosted: Wed Nov 02, 2005 1:00 am    Post subject: Reply with quote

Any progress on communicating this to the Gentoo portage devs?
_________________
Brave Cobra
http://www.bravecobra.com
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sun Nov 27, 2005 2:07 pm    Post subject: Reply with quote

Thanks a lot for the cdb. I saw an obvious increase in portage updates (the 50%-54% slowdown).

Now I'd like to know: should I revert to the original portage behaviour (I probably won't now) would I also have to re-emerge all the packages I installed since I switched to cdb?
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
Chaosite
Guru
Guru


Joined: 13 Dec 2003
Posts: 540
Location: Right over here.

PostPosted: Sun Nov 27, 2005 3:18 pm    Post subject: Reply with quote

VinzC: No.
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sun Nov 27, 2005 4:28 pm    Post subject: Reply with quote

Chaosite wrote:
VinzC: No.

Good. Thanks.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
HAL_9000
Tux's lil' helper
Tux's lil' helper


Joined: 29 Sep 2004
Posts: 114
Location: Netherlands

PostPosted: Mon Nov 28, 2005 3:18 pm    Post subject: Reply with quote

very very very nice :)
_________________
We are the keepers of the sacred words: Ni peng and Nee wom!
Back to top
View user's profile Send private message
fuzzythebear
Guru
Guru


Joined: 28 Nov 2004
Posts: 317

PostPosted: Wed Nov 30, 2005 12:36 pm    Post subject: thanks ... 14 minutes/12 seconds Reply with quote

Great , absolutely great :)

Speedup ? you bet
before 14 minutes after 12 seconds
70 times faster .. on the dot :))

Thank you 70 times :)=

Fuzzy
Back to top
View user's profile Send private message
zigver
Tux's lil' helper
Tux's lil' helper


Joined: 09 May 2003
Posts: 87

PostPosted: Wed Nov 30, 2005 11:41 pm    Post subject: Reply with quote

Can't believe it took me a year to see this post. Simply awesome.
Back to top
View user's profile Send private message
zytek
Apprentice
Apprentice


Joined: 06 Feb 2004
Posts: 233
Location: Poland, Ostrów Wlkp.

PostPosted: Thu Dec 01, 2005 12:47 am    Post subject: Reply with quote

Code:

kanapa edb # time emerge metadata
skipping sync

>>> Updating Portage cache:  100%

real    0m22.034s
user    0m10.521s
sys     0m1.082s



yeeaaah, thank you! :)
Back to top
View user's profile Send private message
chickeneater
n00b
n00b


Joined: 24 Sep 2004
Posts: 23

PostPosted: Thu Dec 01, 2005 1:27 am    Post subject: Reply with quote

I get an error when doing step four: emerge metadata:

Code:
porsche ~ # emerge metadata
skipping sync

>>> Updating Portage cache:       
Failed cache update: app-accessibility/SphinxTrain-0.9.1-r1 Method not defined

Failed cache update: app-accessibility/at-poke-0.2.2 Method not defined

Failed cache update: app-accessibility/brltty-3.6.1-r1 Method not defined

Failed cache update: app-accessibility/brltty-3.6 Method not defined

...
a bunch more of these
...

Traceback (most recent call last):
  File "/usr/bin/emerge", line 2718, in ?
    cleanse_cache(oldcat, savelist)
  File "/usr/bin/emerge", line 2697, in cleanse_cache
    for pv in portage.db["/"]["porttree"].dbapi.auxdb[porttree_root][cat].keys():
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 143, in keys
    self.data.addList[key] = val
NameError: global name 'val' is not defined
Back to top
View user's profile Send private message
bludger
Guru
Guru


Joined: 09 Apr 2003
Posts: 389

PostPosted: Fri Dec 02, 2005 1:53 pm    Post subject: Reply with quote

Just switched over to this and it looks good so far. Although I have been happy with eix up until now, I have been frustrated by the slow speed of equery when searching for the files that belong to a package. Equery is much quicker now.
Back to top
View user's profile Send private message
bludger
Guru
Guru


Joined: 09 Apr 2003
Posts: 389

PostPosted: Fri Dec 02, 2005 1:56 pm    Post subject: Reply with quote

Actually, I take it back. "equery f" to find the contents of a package is much faster now, but "equery b" to find which package a file belongs to is still rather slow. It seems that there is no index in the database on the files.
Back to top
View user's profile Send private message
Cappo
n00b
n00b


Joined: 29 Jun 2004
Posts: 21

PostPosted: Sun Dec 04, 2005 12:50 am    Post subject: Reply with quote

I apologize if this has been answered already in this thread, but I get this error when I try to run emerge metadata:

Code:
Traceback (most recent call last):
  File "/usr/bin/emerge", line 10, in ?
    import portage
  File "/usr/lib/portage/pym/portage.py", line 7140, in ?
    portdb=portdbapi(settings["PORTDIR"])
  File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
    self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
  File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
    self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
  File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
    return load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
    mod = __import__(modname)
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 21, in ?
    import cdb
ImportError: No module named cdb


I think I read before that this is caused if the portage version installed is not recent enough. I have installed:

Code:
curtis linux # emerge -V
Portage 2.0.51.22-r3 (default-linux/x86/no-nptl/2.4, gcc-3.4.4, glibc-2.3.5-r2, 2.4.31-gentoo-r1 i586)
[/code]

Does anyone know what version of portage is necessary?


Thanks,

Cappo
Back to top
View user's profile Send private message
sirdilznik
l33t
l33t


Joined: 28 Apr 2005
Posts: 731

PostPosted: Sun Dec 04, 2005 1:09 am    Post subject: Reply with quote

works like a charm 8)
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sun Dec 04, 2005 9:29 am    Post subject: Reply with quote

Cappo wrote:
I apologize if this has been answered already in this thread, but I get this error when I try to run emerge metadata:

Code:
Traceback (most recent call last):
  File "/usr/bin/emerge", line 10, in ?
    import portage
  File "/usr/lib/portage/pym/portage.py", line 7140, in ?
    portdb=portdbapi(settings["PORTDIR"])
  File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
    self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
  File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
    self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
  File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
    return load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
    mod = __import__(modname)
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 21, in ?
    import cdb
ImportError: No module named cdb

I got the same error message - word upon word - as I had forgotten to emerge python-cdb.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
Cappo
n00b
n00b


Joined: 29 Jun 2004
Posts: 21

PostPosted: Mon Dec 05, 2005 1:01 am    Post subject: Reply with quote

VinzC wrote:
I got the same error message - word upon word - as I had forgotten to emerge python-cdb.


It looks like I have python-cdb installed:

Code:

curtis matt # emerge -pv python-cdb

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild   R   ] dev-python/python-cdb-0.32  18 kB

Total size of downloads: 18 kB

Back to top
View user's profile Send private message
Cappo
n00b
n00b


Joined: 29 Jun 2004
Posts: 21

PostPosted: Thu Dec 08, 2005 2:21 am    Post subject: Reply with quote

I was able to get this working by upgrading to python 3.4 and re-emerging python-cdb. I don't know what the real problem was, but this seemed to fix it.


Cappo
Back to top
View user's profile Send private message
dan2003
Apprentice
Apprentice


Joined: 11 May 2003
Posts: 258
Location: somewhere near london, UK

PostPosted: Sat Dec 17, 2005 12:31 am    Post subject: Reply with quote

I just tried this out and i'm stuck with

Code:
england mango # emerge metadata
Traceback (most recent call last):
  File "/usr/bin/emerge", line 10, in ?
    import portage
  File "/usr/lib/portage/pym/portage.py", line 7140, in ?
    portdb=portdbapi(settings["PORTDIR"])
  File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
    self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
  File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
    self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
  File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
    return load_mod(best_mod)
  File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
    mod = __import__(modname)
  File "/usr/lib/portage/pym/portage_db_cdb.py", line 16, in ?
    USA
NameError: name 'USA' is not defined


I don't know anything about python and so have no clue where to start looking ;(

Help!

Edit : after writing that i decided to look in the file on the off chance, and realised it was a mistake caused with a strange bug i'm experiencing with nano recently
_________________
Too easy!
Back to top
View user's profile Send private message
VinzC
Watchman
Watchman


Joined: 17 Apr 2004
Posts: 5098
Location: Dark side of the mood

PostPosted: Sat Dec 17, 2005 2:36 pm    Post subject: Reply with quote

See this post and the one above.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!
Back to top
View user's profile Send private message
Psiuyo
n00b
n00b


Joined: 22 Apr 2004
Posts: 50
Location: Calgary

PostPosted: Sat Dec 17, 2005 3:22 pm    Post subject: Reply with quote

Also, make sure you are NOT using the newest ~x86 version of portage (available sometime around Dec 16-17th) sys-apps/portage-2.1_pre1, supposedly the "new caching framework which is not backward compatible with older modules".
The long and short of it, remove the cdb lines in /etc/portage/modules

Here is the main thread: https://forums.gentoo.org/viewtopic-t-413535.html
Here is the breakage page: http://dev.gentoo.org/~jstubbs/2.1_pre1-breakage.html
Back to top
View user's profile Send private message
jstubbs
Retired Dev
Retired Dev


Joined: 05 Jul 2003
Posts: 126
Location: Tokyo

PostPosted: Sat Dec 17, 2005 4:59 pm    Post subject: Reply with quote

Psiuyo wrote:
Also, make sure you are NOT using the newest ~x86 version of portage (available sometime around Dec 16-17th) sys-apps/portage-2.1_pre1, supposedly the "new caching framework which is not backward compatible with older modules".
The long and short of it, remove the cdb lines in /etc/portage/modules

Here is the main thread: https://forums.gentoo.org/viewtopic-t-413535.html
Here is the breakage page: http://dev.gentoo.org/~jstubbs/2.1_pre1-breakage.html


I've made a module as noted in my last post to the "main thread". Unsupported though.. I won't be using it anymore. ;)
Back to top
View user's profile Send private message
thoffmeyer
Apprentice
Apprentice


Joined: 11 Apr 2004
Posts: 208
Location: GMT -5 Hours

PostPosted: Sat Dec 17, 2005 9:34 pm    Post subject: Reply with quote

Love it, so much faster thanks.
_________________
Conrad Guide, Current Maintainer

Join us on IRC
Server: irc.freenode.net
Channel: #conrad
Back to top
View user's profile Send private message
_loki_
l33t
l33t


Joined: 18 Dec 2003
Posts: 680
Location: in the shell

PostPosted: Sun Dec 18, 2005 2:40 pm    Post subject: Reply with quote

seems not to work with portage 2.1... that is marked as ~x86 today.. Anyway very nice job!!!
Back to top
View user's profile Send private message
ianegg
Apprentice
Apprentice


Joined: 26 Oct 2005
Posts: 279
Location: Breakfast.

PostPosted: Sun Dec 18, 2005 10:10 pm    Post subject: Reply with quote

Just updated portage without realising it would break this, but the new version (2.1_pre1) seems to update the cache a lot faster, and even seemed a little quicker than this cdb module. No numbers though, sorry.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 6 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum