| View previous topic :: View next topic |
| Author |
Message |
MighMoS
Guru
Joined: 24 Apr 2003
Posts: 416
Location: @ ~
|
 Posted: Thu Mar 10, 2005 3:08 pm Post subject: |
 |
|
Would you mind if I posted this to gentoo-wiki.com?
_________________
jabber: MighMoS@jabber.org
localhost # export HOME=`which heart` |
|
| Back to top |
|
 |
_mikec_
Guru
Joined: 22 May 2004
Posts: 386
|
| Posted: Fri Mar 11, 2005 3:35 am Post subject: |
|
|
not at all, if i there's a fix i want to know about so i can use it, btw i am using reiser4 partition with nitro-sources on amd athlon-xp.
_________________
(8 Mars 2005 - 07:52:46)
(18 September 2017 - 11:25:35) |
|
| Back to top |
|
|
MighMoS
Guru
Joined: 24 Apr 2003
Posts: 416
Location: @ ~
|
| Posted: Sat Mar 12, 2005 2:58 am Post subject: |
|
|
Well, for anyone interested in looking at the wiki article (and feel free to add anything I missed)m its at http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb .
_________________
jabber: MighMoS@jabber.org
localhost # export HOME=`which heart` |
|
| Back to top |
|
|
int2str
Guru
Joined: 29 Sep 2003
Posts: 362
|
| Posted: Sat Mar 12, 2005 4:06 am Post subject: |
|
|
Just another "works for me".... On AMD64 nevertheless.
Here's a mini benchmark:
| Quote: | Before CDB:
emerge -S python
2nd run: 0m16.4
3rd run: 0m14.3
emerge metadata
2nd run: 1m48
3rd run: 1m51
With CDB:
emerge -S python
2nd run: 0m13.4
3rd run: 0m13.9
emerge metadata
2nd run: 0m58
3rd run: 0m49 |
Most importantly (to me), "emerge -s python" (lowercase s) takes just 1 second...
Nice.
One suggestion though:
Put some version information into this script, so we can keep track and see if there's any change.
Cheers,
Andre
_________________
Adpot an unanswered post today! |
|
| Back to top |
|
|
_mikec_
Guru
Joined: 22 May 2004
Posts: 386
|
| Posted: Sat Mar 12, 2005 8:55 pm Post subject: |
|
|
ok it is working now, i dont know what i did different beside using http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb .
emerge sync:
real 1m4.250s
user 0m18.066s
sys 0m4.411s
_________________
(8 Mars 2005 - 07:52:46)
(18 September 2017 - 11:25:35)
Last edited by _mikec_ on Wed Mar 16, 2005 7:20 pm; edited 1 time in total |
|
| Back to top |
|
|
rhill
Retired Dev
Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca
|
| Posted: Tue Mar 15, 2005 10:55 pm Post subject: |
|
|
just wanted to mention that when using this cdb module, upgrading to python 2.4 will break portage until the lines in /etc/portage/modules are commented out. then you can run python-updater to re-emerge any packages built against the old python version (or just emerge python-cdb again) and uncomment the module lines after.
_________________
by design, by neglect
for a fact or just for effect |
|
| Back to top |
|
|
stonent
Veteran
Joined: 07 Aug 2003
Posts: 1139
Location: Texas
|
| Posted: Mon Mar 21, 2005 8:08 am Post subject: |
|
|
| Code: | Aurora / # time esearch python >/dev/null
real 0m0.277s
user 0m0.246s
sys 0m0.025s |
_________________
Inspiron 4100 & Sun UltraAXe
Portage on Solaris|Dell Laptop Hacks
The way you feel about organized religion is the same way I feel about organized socialism. |
|
| Back to top |
|
|
Chaosite
Guru
Joined: 13 Dec 2003
Posts: 540
Location: Right over here.
|
| Posted: Wed Mar 23, 2005 1:47 pm Post subject: |
|
|
| Code: |
Aurora ~ # emerge metadata
>>> Updating Portage cache: 100%
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m50.773s
user 0m9.855s
sys 0m1.420s
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m12.092s
user 0m9.765s
sys 0m0.635s
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m10.672s
user 0m9.765s
sys 0m0.635s
Aurora ~ # mv /etc/portage/modules.back /etc/portage/modules
Aurora ~ # emerge metadata
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m24.370s
user 0m9.855s
sys 0m0.740s
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m10.081s
user 0m9.280s
sys 0m0.560s
Aurora ~ # time emerge world -uDpv > /dev/null
real 0m9.893s
user 0m9.230s
sys 0m0.555s
|
Notice the improvement on the non-cached (AKA, first time...) behaviour. Its pretty good.
Mind you, this is on a pentium 4. The performance margins are better for older machines. |
|
| Back to top |
|
|
Pythonhead
Developer
Joined: 16 Dec 2002
Posts: 1801
Location: Redondo Beach, Republic of Calif.
|
| Posted: Wed Mar 23, 2005 4:25 pm Post subject: |
|
|
f.y.i.:
app-portage/eix versions >0.2.1 don't work with this cdb module. It will search your overlay directories but not PORTDIR. |
|
| Back to top |
|
|
pablo_supertux
Advocate
Joined: 25 Jan 2004
Posts: 2931
Location: Somewhere between reality and Middle-Earth and in Freiburg (Germany)
|
| Posted: Tue Mar 29, 2005 11:22 pm Post subject: |
|
|
Well, I cannot tell that it was really faster. Some nasty examples:
| Code: |
Before:
rex@supertux:~> time emerge w -s >/dev/null
real 0m10.643s
user 0m9.827s
sys 0m0.669s
|
and
| Code: |
after
rex@supertux:~> time emerge w -s >/dev/null
real 0m29.551s
user 0m9.011s
sys 0m0.854s
rex@supertux:~> time emerge w -s >/dev/null
real 0m10.079s
user 0m8.791s
sys 0m0.514s
|
??? The first time took over 19 seconds to complete the search.
| Code: |
before:
rex@supertux:~> time emerge php -s >/dev/null
real 0m11.274s
user 0m2.781s
sys 0m0.351s
rex@supertux:~> time emerge php -s >/dev/null
real 0m3.064s
user 0m2.762s
sys 0m0.181s
|
| Code: |
after
rex@supertux:~> time emerge php -s >/dev/null
real 0m3.471s
user 0m2.617s
sys 0m0.172s
|
_________________
A! Elbereth Gilthoniel!
silivren penna míriel
o menel aglar elenath,
Gilthoniel, A! Elbereth! |
|
| Back to top |
|
|
donno
n00b
Joined: 08 Nov 2004
Posts: 9
|
| Posted: Wed Mar 30, 2005 12:27 am Post subject: |
|
|
| good job. |
|
| Back to top |
|
|
xiaokj
n00b
Joined: 31 Aug 2004
Posts: 25
|
| Posted: Tue Apr 05, 2005 2:37 pm Post subject: |
|
|
I'm interested in using this on my system, but I would like to ask -- which way will be the fastest and safest? The board already includes cdb, mysql and even esearch, even if cdb is the only one described here. May I know what DB is best?
And how about the db in slocate? I liked it. Can a similar implementation be used here? using another language is not really ideal isn't it? not that I don't like it or what.... |
|
| Back to top |
|
|
rowbin hod
n00b
Joined: 08 Jan 2005
Posts: 50
|
| Posted: Tue Apr 05, 2005 7:48 pm Post subject: |
|
|
This is nice 
normal portage:
| Code: |
cube usr # time emerge metadata
skipping sync
>>> Updating Portage cache: 100%
real 4m57.154s
user 2m11.010s
sys 0m17.873s
cube usr #time emerge --searchdesc python
real 0m59.864s
user 0m35.593s
sys 0m6.941s
|
With CDB:
| Code: |
cube portage # time emerge metadata
skipping sync
>>> Updating Portage cache: 100%
real 2m0.692s
user 0m24.563s
sys 0m4.785s
cube usr #time emerge --searchdesc python
real 0m42.652s
user 0m33.966s
sys 0m3.918s
|
So an emerge of the metadata is only 40% of the time it normally is, and the search takes ~70% of the normal time.
Not bad at all for a small tweak 
Athlon XP1600/256DDR |
|
| Back to top |
|
|
drescherjm
Advocate
Joined: 05 Jun 2004
Posts: 2790
Location: Pittsburgh, PA, USA
|
|
| Back to top |
|
|
J.M.I.T.
Guru
Joined: 15 Jan 2003
Posts: 481
Location: München, Germany
|
| Posted: Mon Apr 11, 2005 7:08 pm Post subject: |
|
|
Thanx! Works great on my machine, even if it's a fast one... The db speeds up searches by ~10% according to time...
_________________
Get a life...
Use at own risk... |
|
| Back to top |
|
|
FonderiaDigitale
Veteran
Joined: 06 Nov 2003
Posts: 1710
Location: Rome, Italy
|
| Posted: Wed Apr 13, 2005 7:05 pm Post subject: |
|
|
| Pythonhead wrote: | f.y.i.:
app-portage/eix versions >0.2.1 don't work with this cdb module. It will search your overlay directories but not PORTDIR. |
any way to fix this?
i'm an eix regular, but wanna use also this db.
thanks
_________________
Come disse un amico, i sistemisti sono un po' come gli artigiani per l'informatica
|
|
| Back to top |
|
|
MetX
n00b
Joined: 19 May 2004
Posts: 19
|
| Posted: Sat Apr 16, 2005 8:25 pm Post subject: SQLite for Portage |
|
|
| Leffe wrote: | | Hm, I wonder which is faster SQLite or CDB |
Hmm.. I bet SQLite would be great for this, especially if you set it up to use transactions when updating.
SQLite 'loves' transactions... i forgot exactly why.. but it kicks MYSql's ass when inserting thousands of rows inside a transaction.. ouside.. it doesnt do quite as hot.. but still near MYSql perf.. there is some page out there with the stats... I'm pretty sure SQLite also docs this well. |
|
| Back to top |
|
|
LightEater
n00b
Joined: 10 Oct 2004
Posts: 38
|
| Posted: Fri Apr 22, 2005 11:24 am Post subject: |
|
|
Nice, emerge metadata before
| Quote: | real 2m47.463s
user 1m14.297s
sys 0m19.323s |
and after
| Quote: | real 0m31.356s
user 0m19.787s
sys 0m4.264s |
_________________
I'd like to answer this question if I may in two ways: firstly in my normal voice and then in a kind of silly high-pitched whine. |
|
| Back to top |
|
|
sliwowitz
Apprentice
Joined: 21 Jan 2005
Posts: 215
Location: Europe/Prague
|
| Posted: Tue Jul 26, 2005 8:02 am Post subject: didn't work here |
|
|
I'm getting this error when running emerge
| Code: | gt strazce # emerge metadata
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_cdb.py", line 16, in ?
USA
NameError: name 'USA' is not defined |
I followed the wiki and read this thread through, but can't solve it.
Using anydbm mentioned on this thread's first page works as expected. |
|
| Back to top |
|
|
sn4ip3r
Guru
Joined: 14 Dec 2002
Posts: 325
Location: Tallinn, Estonia
|
| Posted: Tue Jul 26, 2005 11:37 am Post subject: Re: didn't work here |
|
|
| slivovice wrote: | I'm getting this error when running emerge
| Code: | gt strazce # emerge metadata
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_cdb.py", line 16, in ?
USA
NameError: name 'USA' is not defined |
I followed the wiki and read this thread through, but can't solve it.
Using anydbm mentioned on this thread's first page works as expected. |
You most likely copied the code badly, notice that the last line in the header comment is "# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA", the "USA" was probably misplaced on the next line. |
|
| Back to top |
|
|
Jerem
Apprentice
Joined: 11 Jun 2004
Posts: 177
|
| Posted: Tue Jul 26, 2005 9:33 pm Post subject: |
|
|
Good work but there is nothing faster than eix to perform description searches.
app-portage/eix is even faster than portage-c...
Anyway, that python script is great, and deserves to be put in official portage.
Thanks a lot ! |
|
| Back to top |
|
|
sliwowitz
Apprentice
Joined: 21 Jan 2005
Posts: 215
Location: Europe/Prague
|
| Posted: Tue Aug 02, 2005 1:47 pm Post subject: /me - dumb |
|
|
thanks sn4ip3r nano cut my line off. what a dumb negligence of me  |
|
| Back to top |
|
|
nxsty
Veteran
Joined: 23 Jun 2004
Posts: 1556
Location: .se
|
| Posted: Tue Aug 02, 2005 2:26 pm Post subject: |
|
|
Does this also work for the latest portage 2.0.51.22-r2?
@tobidope: Why don't you add this to a enhancement bug-report against portage? |
|
| Back to top |
|
|
sn4ip3r
Guru
Joined: 14 Dec 2002
Posts: 325
Location: Tallinn, Estonia
|
| Posted: Tue Aug 02, 2005 2:29 pm Post subject: |
|
|
| nxsty wrote: | Does this also work for the latest portage 2.0.51.22-r2?
@tobidope: Why don't you add this to a enhancement bug-report against portage? |
I think it does, haven't seen any errors, but I also haven't done any benchmarks. |
|
| Back to top |
|
|
Klejs
n00b
Joined: 29 Jul 2005
Posts: 27
Location: Norrhult, Sweden
|
| Posted: Tue Aug 02, 2005 4:25 pm Post subject: |
|
|
| REALLY nice! |
|
| Back to top |
|
|
|
Unsupported Software
