View previous topic :: View next topic |
Author |
Message |
alec Apprentice
Joined: 19 Apr 2002 Posts: 270 Location: Here
|
Posted: Fri Mar 26, 2004 9:43 pm Post subject: [TIP] Using psyco to speed up emerge |
|
|
This tip was on the gentoo-user list a day or so ago; credit goes to "Gregory P. Smith".
Quote: | I was recently playing with psyco (the x86 jit-like compiler for python)
i figured i'd try out emerge with it to see what effect it had.
emerge is largely IO bound so for a first run to load the buffer cache
the effect isn't likely to be noticed either way.
however on later runs once the buffer cache was primed i found that the
use of psyco in emerge sped up "emerge search xfs" by >20% and sped up
"emerge -pU world" by ~35%. I tested using psyco 1.2 on a celery333.
I recommend adding the following to emerge:
|
Code: | try:
import psyco
psyco.full()
except ImportError:
pass
|
Quote: |
Anyone who has installed psyco (its an ebuild) will benefit from a faster
searching and dependancy calculations.
I also watched the memory usage during the runs as well. emerge without
psyco consumed 7.5-10mb of ram. emerge with psync consumed 9-13mb.
|
Emerge 'feels' faster for me; one or two others posted similar results on the list. No one has posted difficulties.
Edit: Please see the post two or three down about where to put this in '/usr/bin/emerge'. This will only speed up the program 'emerge' itself, not compiles. You'll need to do this again if you ever upgrade portage. Additionally, it isn't quite clear in his message, but you need to 'emerge psyco' to use this.
Last edited by alec on Fri Mar 26, 2004 10:02 pm; edited 1 time in total |
|
Back to top |
|
|
neenee Veteran
Joined: 20 Jul 2003 Posts: 1786
|
Posted: Fri Mar 26, 2004 9:54 pm Post subject: |
|
|
perhaps a silly question - but where would i add that bit?
to the end of /usr/bin/emerge? |
|
Back to top |
|
|
ed0n l33t
Joined: 23 Apr 2003 Posts: 638 Location: Prishtine/Kosove
|
Posted: Fri Mar 26, 2004 9:57 pm Post subject: |
|
|
neenee wrote: | perhaps a silly question - but where would i add that bit?
to the end of /usr/bin/emerge? |
I wanted to ask that too. |
|
Back to top |
|
|
alec Apprentice
Joined: 19 Apr 2002 Posts: 270 Location: Here
|
Posted: Fri Mar 26, 2004 9:57 pm Post subject: |
|
|
I don't know the first thing about python, but I added it right after the initial import, so my /usr/bin/emerge looks like this (from the top):
Code: | #!/usr/bin/python -O
# Copyright 1999-2003 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2
# $Header: /home/cvsroot/gentoo-src/portage/bin/emerge,v 1.264 2004/02/08 21:24:15 nakano Exp $
import os,sys
os.environ["PORTAGE_CALLER"]="emerge"
sys.path = ["/usr/lib/portage/pym"]+sys.path
try:
import psyco
psyco.full()
except ImportError:
pass
import emergehelp,xpak,string,re,commands,time,shutil,traceback,atexit,signal,socket,types
from stat import *
from output import *
|
|
|
Back to top |
|
|
neenee Veteran
Joined: 20 Jul 2003 Posts: 1786
|
Posted: Fri Mar 26, 2004 10:10 pm Post subject: |
|
|
thanks for the prompt reply |
|
Back to top |
|
|
cbr Apprentice
Joined: 05 Jan 2004 Posts: 285 Location: Tallinn/Rakvere, Estonia
|
Posted: Sat Mar 27, 2004 12:04 am Post subject: |
|
|
OMG
This speeds up searching A LOT!
Thanks for the tip |
|
Back to top |
|
|
discomfitor l33t
Joined: 21 Feb 2003 Posts: 927 Location: None
|
Posted: Sat Mar 27, 2004 12:20 am Post subject: |
|
|
WOW. That's amazing! It fails horribly. Segfaults everywhere. _________________ There is no substitute for experience.
Imperfection indicates a lack of effort. |
|
Back to top |
|
|
alec Apprentice
Joined: 19 Apr 2002 Posts: 270 Location: Here
|
Posted: Sat Mar 27, 2004 2:14 am Post subject: |
|
|
Well, I suppose YMMV :)
I've been using it for a day on an ~x86 and a x86 system, and have had no problems. |
|
Back to top |
|
|
snakattak3 Guru
Joined: 11 Dec 2002 Posts: 468 Location: Seattle
|
Posted: Sat Mar 27, 2004 4:58 am Post subject: |
|
|
WOW. Searching is almost instantaneous! Even "emerge -p depclean" is much faster. It used to take almost a full minute sometimes. That was one thing that always bugged me about portage, slow on searching.
We should have a USE="psyco" flag, that way it can patch portage (/usr/bin/emerge) everytime you upgrade or install it. That wouldn't be too hard would it? _________________ Ban Reality TV!
Adopt an Unanswered Post |
|
Back to top |
|
|
LAsk n00b
Joined: 02 Dec 2003 Posts: 32 Location: Avesta, Sweden
|
Posted: Sat Mar 27, 2004 10:16 am Post subject: |
|
|
It fails for me too
These are the errors when I do a emerge world -p
Code: | /usr/lib/portage/pym/portage.py:2952: warning: eval()/execfile() cannot see the locals in functions bound by Psyco; consider using eval() in its two- or three-arguments form
def match2(self,mydep,mykey,mylist):
Traceback (most recent call last):
File "/usr/bin/emerge", line 2176, in ?
if not mydepgraph.xcreate(myaction):
File "/usr/bin/emerge", line 1042, in xcreate
if not self.create(myk):
File "/usr/bin/emerge", line 755, in create
if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
File "/usr/bin/emerge", line 945, in select_dep
if not self.create(myk,myparent,myuse=binpkguseflags):
File "/usr/bin/emerge", line 755, in create
if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
File "/usr/bin/emerge", line 877, in select_dep
myeb=portage.portdb.xmatch("bestmatch-visible",x)
File "/usr/lib/portage/pym/portage.py", line 4065, in xmatch
myval=best(self.xmatch("match-visible",None,mydep,mykey))
File "/usr/lib/portage/pym/portage.py", line 4076, in xmatch
myval=self.match2(mydep,mykey,self.xmatch("list-visible",None,mydep,mykey))
File "/usr/lib/portage/pym/portage.py", line 3017, in match2
if eval("pkgcmp(cp_x[1:],cp_key[1:])"+cmpstr+"0"):
File "<string>", line 0, in ?
NameError: name 'cp_x' is not defined
|
Anyone who knows what it is that might be wrong? _________________ Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgement.
- Tolkien |
|
Back to top |
|
|
AlterEgo Veteran
Joined: 25 Apr 2002 Posts: 1619
|
Posted: Sat Mar 27, 2004 10:40 am Post subject: |
|
|
Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.
Not that amazing |
|
Back to top |
|
|
ed0n l33t
Joined: 23 Apr 2003 Posts: 638 Location: Prishtine/Kosove
|
Posted: Sat Mar 27, 2004 1:19 pm Post subject: |
|
|
AlterEgo wrote: | Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.
Not that amazing |
I got that result too. |
|
Back to top |
|
|
Vagabond Apprentice
Joined: 19 Jan 2003 Posts: 192
|
Posted: Sat Mar 27, 2004 2:25 pm Post subject: |
|
|
you guys must have slow computers, I get 10 seconds for vanilla emerge -pu world and 8 seconds with psyco. And my portage tree is even mounted over NFS
Vag |
|
Back to top |
|
|
øxygen Apprentice
Joined: 09 Mar 2004 Posts: 236 Location: Bergheim, Germany
|
Posted: Sat Mar 27, 2004 2:50 pm Post subject: |
|
|
or you have a small world file... btw: works |
|
Back to top |
|
|
kronon Apprentice
Joined: 11 Aug 2003 Posts: 212 Location: NL
|
Posted: Sat Mar 27, 2004 4:19 pm Post subject: |
|
|
some notes to get it working.
1).you have to emerge psyco first.
2).python is to damn sensetive
this code will work
try:
import psyco
psyco.full()
except ImportError:
pass
NOTE: only difference is the space before except, this dit the trick for me. |
|
Back to top |
|
|
neenee Veteran
Joined: 20 Jul 2003 Posts: 1786
|
Posted: Sat Mar 27, 2004 10:14 pm Post subject: |
|
|
consider posting that bit between code tags.
unless it's meant to have no space before any
of the lines. |
|
Back to top |
|
|
LAsk n00b
Joined: 02 Dec 2003 Posts: 32 Location: Avesta, Sweden
|
Posted: Sat Mar 27, 2004 10:15 pm Post subject: |
|
|
kronon wrote: | some notes to get it working.
1).you have to emerge psyco first.
2).python is to damn sensetive
this code will work
try:
import psyco
psyco.full()
except ImportError:
pass
NOTE: only difference is the space before except, this dit the trick for me. |
Still the same problem here _________________ Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgement.
- Tolkien |
|
Back to top |
|
|
mastergoon Apprentice
Joined: 27 Jul 2003 Posts: 161 Location: Portland, OR
|
Posted: Sun Mar 28, 2004 12:45 am Post subject: |
|
|
make sure you use real tabs to indent in emerge |
|
Back to top |
|
|
PovMan Guru
Joined: 30 Sep 2002 Posts: 375 Location: Australia
|
Posted: Sun Mar 28, 2004 3:14 am Post subject: |
|
|
For anyone who's having segfaults, make sure you do an 'emerge psyco' before modifying emerge
It works great for me! Why isn't this part of emerge already? (like as a use flag) _________________ I am SPAM, hear me roar.
Before posting your own topic, try to answer at least one unanswered one. |
|
Back to top |
|
|
verbatim Apprentice
Joined: 13 Mar 2003 Posts: 223
|
Posted: Sun Mar 28, 2004 3:42 am Post subject: |
|
|
Hey, works for me... getting a measureable speedup.
user time for emerge -epv world without: 33 seconds
with: 22 seconds
Cool. |
|
Back to top |
|
|
schism39401 Tux's lil' helper
Joined: 13 Mar 2003 Posts: 130
|
Posted: Sun Mar 28, 2004 5:12 am Post subject: |
|
|
Searching does seem a little faster...I use esearch usually and it's a lot faster |
|
Back to top |
|
|
djkork n00b
Joined: 25 May 2004 Posts: 45 Location: Madrid
|
Posted: Tue May 25, 2004 1:40 am Post subject: |
|
|
this does not speed up too much....
emerge -puD world
20 sec with psico
18 sec with psico |
|
Back to top |
|
|
gbu n00b
Joined: 18 May 2004 Posts: 25 Location: Between UCI and UCLA
|
Posted: Tue May 25, 2004 10:14 pm Post subject: |
|
|
Of course you can do the same thing to esearch and eupdatedb. |
|
Back to top |
|
|
equilibrium Apprentice
Joined: 29 Jun 2003 Posts: 213 Location: UK
|
Posted: Wed May 26, 2004 11:48 am Post subject: |
|
|
omg nice
searching seems faster will see how fast it is to emerge some stuff in future. Am thinking of upgrading KDE _________________ kernel 4.15.17-1-equk | i3wm | github |
|
Back to top |
|
|
neenee Veteran
Joined: 20 Jul 2003 Posts: 1786
|
Posted: Wed May 26, 2004 12:09 pm Post subject: |
|
|
emerge -uDpv world,
without psyco:
Code: | real 0m7.071s
user 0m5.472s
sys 0m0.631s
|
with psyco:
Code: | real 0m4.572s
user 0m3.420s
sys 0m0.648s |
ps. i only timed the second and third time, to be sure the difference
in time was not because of the files/process being cached. |
|
Back to top |
|
|
|