[TIP] Using psyco to speed up emerge

Message

alec · Post by **alec** » Fri Mar 26, 2004 9:43 pm

This tip was on the gentoo-user list a day or so ago; credit goes to "Gregory P. Smith".

I was recently playing with psyco (the x86 jit-like compiler for python)
i figured i'd try out emerge with it to see what effect it had.

emerge is largely IO bound so for a first run to load the buffer cache
the effect isn't likely to be noticed either way.

however on later runs once the buffer cache was primed i found that the
use of psyco in emerge sped up "emerge search xfs" by >20% and sped up
"emerge -pU world" by ~35%. I tested using psyco 1.2 on a celery333.

I recommend adding the following to emerge:

Code: Select all

try:
    import psyco
    psyco.full()
except ImportError:
    pass

Anyone who has installed psyco (its an ebuild) will benefit from a faster
searching and dependancy calculations.

I also watched the memory usage during the runs as well. emerge without
psyco consumed 7.5-10mb of ram. emerge with psync consumed 9-13mb.

Emerge 'feels' faster for me; one or two others posted similar results on the list. No one has posted difficulties.

Edit: Please see the post two or three down about where to put this in '/usr/bin/emerge'. This will only speed up the program 'emerge' itself, not compiles. You'll need to do this again if you ever upgrade portage. Additionally, it isn't quite clear in his message, but you need to 'emerge psyco' to use this.

neenee · Post by **neenee** » Fri Mar 26, 2004 9:54 pm

perhaps a silly question - but where would i add that bit?

to the end of /usr/bin/emerge?

ed0n · Post by **ed0n** » Fri Mar 26, 2004 9:57 pm

neenee wrote:perhaps a silly question - but where would i add that bit?

to the end of /usr/bin/emerge?

I wanted to ask that too.

alec · Post by **alec** » Fri Mar 26, 2004 9:57 pm

I don't know the first thing about python, but I added it right after the initial import, so my /usr/bin/emerge looks like this (from the top):

Code: Select all

#!/usr/bin/python -O
# Copyright 1999-2003 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2
# $Header: /home/cvsroot/gentoo-src/portage/bin/emerge,v 1.264 2004/02/08 21:24:15 nakano Exp $
                                                                                                                               
import os,sys
os.environ["PORTAGE_CALLER"]="emerge"
sys.path = ["/usr/lib/portage/pym"]+sys.path
                                                                                                                               
try:
        import psyco
        psyco.full()
except ImportError:
        pass
                                                                                                                               
import emergehelp,xpak,string,re,commands,time,shutil,traceback,atexit,signal,socket,types
from stat import *
from output import *

neenee · Post by **neenee** » Fri Mar 26, 2004 10:10 pm

thanks for the prompt reply

cbr · Post by **cbr** » Sat Mar 27, 2004 12:04 am

OMG

This speeds up searching A LOT!
Thanks for the tip

discomfitor · Post by **discomfitor** » Sat Mar 27, 2004 12:20 am

WOW. That's amazing! It fails horribly. Segfaults everywhere.

alec · Post by **alec** » Sat Mar 27, 2004 2:14 am

Well, I suppose YMMV :)

I've been using it for a day on an ~x86 and a x86 system, and have had no problems.

snakattak3 · Post by **snakattak3** » Sat Mar 27, 2004 4:58 am

WOW. Searching is almost instantaneous! Even "emerge -p depclean" is much faster. It used to take almost a full minute sometimes. That was one thing that always bugged me about portage, slow on searching.
We should have a USE="psyco" flag, that way it can patch portage (/usr/bin/emerge) everytime you upgrade or install it. That wouldn't be too hard would it?

LAsk · Post by **LAsk** » Sat Mar 27, 2004 10:16 am

It fails for me too

These are the errors when I do a emerge world -p

Code: Select all

/usr/lib/portage/pym/portage.py:2952: warning: eval()/execfile() cannot see the locals in functions bound by Psyco; consider using eval() in its two- or three-arguments form
  def match2(self,mydep,mykey,mylist):
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2176, in ?
    if not mydepgraph.xcreate(myaction):
  File "/usr/bin/emerge", line 1042, in xcreate
    if not self.create(myk):
  File "/usr/bin/emerge", line 755, in create
    if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
  File "/usr/bin/emerge", line 945, in select_dep
    if not self.create(myk,myparent,myuse=binpkguseflags):
  File "/usr/bin/emerge", line 755, in create
    if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
  File "/usr/bin/emerge", line 877, in select_dep
    myeb=portage.portdb.xmatch("bestmatch-visible",x)
  File "/usr/lib/portage/pym/portage.py", line 4065, in xmatch
    myval=best(self.xmatch("match-visible",None,mydep,mykey))
  File "/usr/lib/portage/pym/portage.py", line 4076, in xmatch
    myval=self.match2(mydep,mykey,self.xmatch("list-visible",None,mydep,mykey))
  File "/usr/lib/portage/pym/portage.py", line 3017, in match2
    if eval("pkgcmp(cp_x[1:],cp_key[1:])"+cmpstr+"0"):
  File "<string>", line 0, in ?
NameError: name 'cp_x' is not defined

Anyone who knows what it is that might be wrong?

AlterEgo · Post by **AlterEgo** » Sat Mar 27, 2004 10:40 am

Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing

ed0n · Post by **ed0n** » Sat Mar 27, 2004 1:19 pm

AlterEgo wrote:Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing

I got that result too.

Vagabond · Post by **Vagabond** » Sat Mar 27, 2004 2:25 pm

you guys must have slow computers, I get 10 seconds for vanilla emerge -pu world and 8 seconds with psyco. And my portage tree is even mounted over NFS

Vag

øxygen · Post by **øxygen** » Sat Mar 27, 2004 2:50 pm

or you have a small world file... btw: works

kronon · Post by **kronon** » Sat Mar 27, 2004 4:19 pm

some notes to get it working.

1).you have to emerge psyco first.
2).python is to damn sensetive

this code will work

try:
import psyco
psyco.full()
except ImportError:
pass

NOTE: only difference is the space before except, this dit the trick for me.

neenee · Post by **neenee** » Sat Mar 27, 2004 10:14 pm

consider posting that bit between code tags.

unless it's meant to have no space before any
of the lines.

LAsk · Post by **LAsk** » Sat Mar 27, 2004 10:15 pm

kronon wrote:some notes to get it working.

1).you have to emerge psyco first.
2).python is to damn sensetive

this code will work

try:
import psyco
psyco.full()
except ImportError:
pass

NOTE: only difference is the space before except, this dit the trick for me.

Still the same problem here

mastergoon · Post by **mastergoon** » Sun Mar 28, 2004 12:45 am

make sure you use real tabs to indent in emerge

PovMan · Post by **PovMan** » Sun Mar 28, 2004 3:14 am

For anyone who's having segfaults, make sure you do an 'emerge psyco' before modifying emerge
It works great for me! Why isn't this part of emerge already? (like as a use flag)

verbatim · Post by **verbatim** » Sun Mar 28, 2004 3:42 am

Hey, works for me... getting a measureable speedup.

user time for emerge -epv world without: 33 seconds
with: 22 seconds

Cool.

schism39401 · Post by **schism39401** » Sun Mar 28, 2004 5:12 am

Searching does seem a little faster...I use esearch usually and it's a lot faster

djkork · Post by **djkork** » Tue May 25, 2004 1:40 am

this does not speed up too much....

emerge -puD world

20 sec with psico
18 sec with psico

gbu · Post by **gbu** » Tue May 25, 2004 10:14 pm

Of course you can do the same thing to esearch and eupdatedb.

equilibrium · Post by **equilibrium** » Wed May 26, 2004 11:48 am

omg nice

searching seems faster

will see how fast it is to emerge some stuff in future. Am thinking of upgrading KDE

neenee · Post by **neenee** » Wed May 26, 2004 12:09 pm

emerge -uDpv world,

without psyco:

Code: Select all

real    0m7.071s
user    0m5.472s
sys     0m0.631s

with psyco:

Code: Select all

real    0m4.572s
user    0m3.420s
sys     0m0.648s

ps. i only timed the second and third time, to be sure the difference
in time was not because of the files/process being cached.