View previous topic :: View next topic |
Author |
Message |
lavacano Apprentice
Joined: 29 May 2005 Posts: 190 Location: Poulsbo, WA
|
Posted: Tue Aug 02, 2005 4:38 pm Post subject: |
|
|
yeah this rocks the box. I saw a dev on the bug ticket take real interest, maybe there is a possibility for this to be default? _________________ Sincerely,
Chadwick Ferguson |
|
Back to top |
|
|
gwolf Guru
Joined: 05 May 2005 Posts: 374 Location: Estonia
|
Posted: Wed Aug 03, 2005 8:35 am Post subject: |
|
|
Really good job!
Note that this script also speeds up equery (a part of gentoolkit). _________________ War is peace. Freedom is slavery. Ketchup is a vegetable. |
|
Back to top |
|
|
burn-gentoo n00b
Joined: 17 Jun 2004 Posts: 13
|
Posted: Tue Sep 06, 2005 4:11 pm Post subject: |
|
|
I decided to do some benchmarks on this handy mod, on multiple systems to get an idea of how well it actually speeds up the portage cache. This is what I've come up with so far. Please keep in mind that these systems are hugely different in terms of RAM and disk type/configuration.
Test Conditions on all systems
Code: | start with 0.00 load
15 minute lapse between each command for memory clearing
same timestamp for portage (equal files) |
System specs are as follows:
TEST1 OS: Linux 2.6.11-hardened-r15/i686 - Distro: Gentoo 1.6.13 - CPU: 2 x Intel Pentium 4 (2992.845 MHz) - Processes: 111 - Memory Usage: 385.70MB/3026.82MB (12.74%) - NOTE: is using md raid 1 (disk writes will be pokey, disk reads will be quite fast)
TEST2 OS: Linux 2.6.12-gentoo-r6/i686 - Distro: Gentoo 1.6.13 - CPU: 2 x Intel Pentium 4 (2599.028 MHz) - Processes: 63 - Memory Usage: 198.92MB/1010.88MB (19.68%)
TEST3 OS: Linux 2.6.12-gentoo-r9/i686 - Distro: Gentoo 1.6.13 - CPU: Pentium III (Coppermine) (737.194 MHz) - Processes: 62 - Memory Usage: 58.54MB/122.71MB (47.70%)
Without CDB - regular ole portage
Code: | TEST1
# time emerge metadata
real 0m50.483s
# time emerge -s python
real 0m1.457s
TEST2
# time emerge metadata
real 0m15.253s
# time emerge -s python
real 0m1.499s
TEST3
# time emerge metadata
real 9m17.262s
# time emerge -s python
real 0m49.744s |
With CDB portage mod - (keep in mind that the emerge metadata test is done with an UPDATE to the portage cache. The CDB files have already been built, timing the initial build process is pointless)
Code: | TEST1
# time emerge metadata
real 0m13.186s
# time emerge -s python
real 0m1.368s
TEST2
# time emerge metadata
real 0m13.923s
# time emerge -s python
real 0m1.325s
TEST3
# time emerge metadata
real 1m47.600s
# time emerge -s python
real 0m13.332s |
Looks to me like it makes very little difference on newer systems, but damn, what a difference on the mothball in there. |
|
Back to top |
|
|
gwolf Guru
Joined: 05 May 2005 Posts: 374 Location: Estonia
|
Posted: Wed Sep 07, 2005 2:08 pm Post subject: |
|
|
Just wondering when will it be compatible with eix? _________________ War is peace. Freedom is slavery. Ketchup is a vegetable. |
|
Back to top |
|
|
appro Tux's lil' helper
Joined: 21 Jul 2003 Posts: 144 Location: Germany/Rosenheim
|
Posted: Sat Sep 10, 2005 5:53 pm Post subject: |
|
|
eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code: | PORTDIR_CACHE_METHOD="cdb" |
into your /etc/eixrc. _________________ If a packet hits a pocket on a socket on a port, and the bus is interrupted and the interrupt's not caught, then the socket packet pocket has an error to report. |
|
Back to top |
|
|
gwolf Guru
Joined: 05 May 2005 Posts: 374 Location: Estonia
|
Posted: Sat Sep 10, 2005 7:23 pm Post subject: |
|
|
appro wrote: | eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code: | PORTDIR_CACHE_METHOD="cdb" |
into your /etc/eixrc. |
And it really was that simple!
Big thanks! _________________ War is peace. Freedom is slavery. Ketchup is a vegetable. |
|
Back to top |
|
|
whitesouls Guru
Joined: 19 Nov 2004 Posts: 358 Location: In Front of My Laptop
|
Posted: Sat Sep 10, 2005 8:06 pm Post subject: |
|
|
appro wrote: | eix can read the cache generated by cdb since eix-0.2.6.
You'll just need to put
Code: | PORTDIR_CACHE_METHOD="cdb" |
into your /etc/eixrc. |
Thanks appro, it worked for me well too.. thank you tobidope for the CDB. _________________ whitesouls
Please insert the [SOLVED] tag if your problem is solved in your respective thread. |
|
Back to top |
|
|
rhill Retired Dev
Joined: 22 Oct 2004 Posts: 1629 Location: sk.ca
|
Posted: Sun Sep 11, 2005 7:27 am Post subject: |
|
|
it still makes a difference for me at least.
Code: | Family: 15 Model: 4 Stepping: 1 Type: 0 Brand: 0
CPU Model: Pentium 4 (Prescott) [E0]
model name : Mobile Intel(R) Pentium(R) 4 CPU 3.06GHz
Feature flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflsh ds acpi mmx fxsr sse sse2 ss ht tm pbe sse3 monitor ds-cpl est tm2 cntx-id xTPR |
HD is a slow as hell 4200RPM 80GiB laptop drive.
default
Code: | # rm -r /var/cache/edb
# time emerge metadata
real 1m57.441s
user 1m17.038s
sys 0m10.791s |
cdb
Code: | # rm -r /var/cache/edb
# time emerge metadata
real 0m37.501s
user 0m18.957s
sys 0m2.869s |
_________________ by design, by neglect
for a fact or just for effect |
|
Back to top |
|
|
hongqn n00b
Joined: 04 Apr 2005 Posts: 22 Location: Beijing, China
|
Posted: Tue Sep 13, 2005 8:01 am Post subject: |
|
|
It looks nice
However, I wander what else will this tip benefit me except the speeding up of portage search? I always use eix to search packages and am satisfied with its perfermance. _________________ God's in his heaven, all's right with the world.
Registered Linux User #396996 |
|
Back to top |
|
|
bob_111 Apprentice
Joined: 12 Oct 2004 Posts: 155
|
Posted: Tue Sep 13, 2005 12:17 pm Post subject: |
|
|
WOW! I just installed this and i am amazed! My PC is rellativly fast and now i can search almost instantly! Nice work! I don't see why this shouldn't be added to regular portage!
- bob_111 |
|
Back to top |
|
|
crs Apprentice
Joined: 19 Apr 2003 Posts: 188
|
Posted: Wed Sep 14, 2005 9:30 am Post subject: |
|
|
Hello. I have problem with newest portage version (Portage 1.589-cvs). Its from ebuild: 2.1.0_alpha20050718.
If i have turn on cdb module i get errors:
Code: | # emerge metadata
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 4997, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 2729, in __init__
gid=portage_gid)
TypeError: __init__() got an unexpected keyword argument 'basepath'
|
And can't do anything with portage because all the time get this error.
cdb module don't support 2.1.* portage version? |
|
Back to top |
|
|
Gentii Guru
Joined: 01 Feb 2004 Posts: 306
|
Posted: Sat Sep 24, 2005 11:41 am Post subject: |
|
|
Well, I've troubles with sys-apps/portage-2.0.52-r1 as well.
Code: | # emerge metadata
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7149, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5134, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4977, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1226, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_cdb.py", line 16, in ?
USA
NameError: name 'USA' is not defined
|
|
|
Back to top |
|
|
sn4ip3r Guru
Joined: 14 Dec 2002 Posts: 325 Location: Tallinn, Estonia
|
Posted: Sat Sep 24, 2005 12:09 pm Post subject: |
|
|
Gentii wrote: | Well, I've troubles with sys-apps/portage-2.0.52-r1 as well.
... |
Already solved. |
|
Back to top |
|
|
Gentree Watchman
Joined: 01 Jul 2003 Posts: 5350 Location: France, Old Europe
|
Posted: Mon Oct 17, 2005 10:08 pm Post subject: |
|
|
Oouustanding !
sync no longer takes 3minutes to do the 52nd percentile and searches are so much quicker it's unnerving.
A great contribution. Many thanks. _________________ Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
Back to top |
|
|
wangxiaohu Apprentice
Joined: 15 Nov 2003 Posts: 248
|
Posted: Tue Oct 18, 2005 9:49 pm Post subject: |
|
|
Following the intructions, I switched portage to cdb based. Now it is possible to use SQLite as a backend? |
|
Back to top |
|
|
sn4ip3r Guru
Joined: 14 Dec 2002 Posts: 325 Location: Tallinn, Estonia
|
Posted: Tue Oct 18, 2005 10:51 pm Post subject: |
|
|
wangxiaohu wrote: | Following the intructions, I switched portage to cdb based. Now it is possible to use SQLite as a backend? |
It sure is, if you write the code. |
|
Back to top |
|
|
wyv3rn Apprentice
Joined: 18 Aug 2005 Posts: 154 Location: USA
|
Posted: Sat Oct 22, 2005 7:44 pm Post subject: |
|
|
Anyone benchmark this against anydbm? |
|
Back to top |
|
|
wyv3rn Apprentice
Joined: 18 Aug 2005 Posts: 154 Location: USA
|
Posted: Sat Oct 22, 2005 9:59 pm Post subject: |
|
|
Machine:
Intel P3 Celeron 800Mhz/100Mhz FSB
256MB PC100 CAS 2 RAM
Adaptec 39160 U160 SCSI Controller
Fujitsu MAN3184 10,000RPM SCSI HD
Test methadology:
cd /var/cache/edb/dep/usr/portage ; rm -rf *
change module to be tested (cpickle, anydbm, cdb) in /etc/portage/modules
reboot, then run the following tests:
TEST1:
time emerge metadata
time emerge metadata
TEST2:
time emerge -upDN1 world
TEST3:
time emerge --searchdesc mozilla
time emerge search python
Results:
--- CPICKLE ---
TEST 1:
real 12m20.934s
user 7m52.709s
sys 0m50.588s
real 1m48.220s
user 0m46.159s
sys 0m10.355s
TEST 2:
real 0m18.804s
user 0m11.656s
sys 0m1.270s
TEST 3:
real 1m47.575s
user 0m56.247s
sys 0m10.196s
real 0m8.594s
user 0m3.647s
sys 0m0.617s
----------------------
--- ANYDBM ---
TEST 1:
real 8m34.878s
user 1m9.006s
sys 0m19.395s
real 2m18.461s
user 0m54.045s
sys 0m7.589s
TEST 2:
real 0m25.021s
user 0m13.158s
sys 0m1.146s
TEST 3:
real 2m0.898s
user 1m4.922s
sys 0m7.351s
real 0m15.963s
user 0m3.537s
sys 0m0.755s
--------------------
--- CDB ---
TEST 1:
real 3m9.459s
user 0m54.717s
sys 0m11.465s
real 1m1.524s
user 0m41.711s
sys 0m3.901s
TEST 2:
real 0m14.951s
user 0m10.903s
sys 0m0.718s
TEST 3:
real 1m8.174s
user 0m52.572s
sys 0m3.704s
real 0m4.272s
user 0m3.223s
sys 0m0.381s
-------------------- |
|
Back to top |
|
|
smithjd15 Tux's lil' helper
Joined: 12 Mar 2003 Posts: 130 Location: Edmonton, AB, CA
|
Posted: Sun Oct 23, 2005 5:19 pm Post subject: |
|
|
I had issues with the version of this script on the initial post. The one on gentoo-wiki worked great. Running them both through Kompare showed extra whitespace on the copy-and-pasted version from this thread that was not there in the wiki version.
If you are having issues with the version in this thread, use the version at the wiki. http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb .
Perhaps the version in this thread can be fixed from the wiki version? |
|
Back to top |
|
|
gordin Guru
Joined: 11 Oct 2002 Posts: 300 Location: Germany/WI
|
Posted: Tue Oct 25, 2005 11:35 pm Post subject: |
|
|
works great on my iBook G3...
updating portage cache took about 45min before... that was really annoying
Now it is taking about 5min... thats a time I can live with. |
|
Back to top |
|
|
matobsk n00b
Joined: 16 Oct 2005 Posts: 64
|
Posted: Wed Oct 26, 2005 5:10 am Post subject: |
|
|
Thank you for finally alleviating the major slowdown at the 50-52% of regenerating my cache.
No matter what I did, regular portage would speed right along to 50%, then use 100% cpu for 8-10 minutes to get to 53%, then fly along again.
This method eliminated that bottleneck, and made the whole portage system faster too. |
|
Back to top |
|
|
R. Daneel Olivaw n00b
Joined: 06 Jul 2004 Posts: 47
|
Posted: Wed Oct 26, 2005 7:53 pm Post subject: Re: /me - dumb |
|
|
slivovice wrote: | thanks sn4ip3r nano cut my line off. what a dumb negligence of me |
IMHO that's more a problem of nano default behaviour which is dumb... |
|
Back to top |
|
|
fourteen20 n00b
Joined: 15 Aug 2003 Posts: 36 Location: Cornwall,uk
|
Posted: Thu Oct 27, 2005 1:52 am Post subject: |
|
|
Thanks to all for this great portage modification, emerge sync was starting to annoy me with the time it was taking _________________ I Wanna be a Guru |
|
Back to top |
|
|
dirtbag Guru
Joined: 18 Feb 2003 Posts: 508 Location: NC
|
Posted: Mon Oct 31, 2005 11:34 pm Post subject: grrrrr |
|
|
ive been running this on my system and I just updated portage and did
emerge update system and I believe it updated python..
now when I try emerges, I get
dellbox jason # emerge portage
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_cdb.py", line 21, in ?
import cdb
ImportError: No module named cdb
dellbox jason #
any ideas? |
|
Back to top |
|
|
pussi l33t
Joined: 08 May 2004 Posts: 727 Location: Finland
|
Posted: Tue Nov 01, 2005 12:21 am Post subject: |
|
|
Run this as root:
Code: | mv /etc/portage/modules /etc/portage/__modules
python-updater
mv /etc/portage/__modules /etc/portage/modules |
|
|
Back to top |
|
|
|