Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP] Using psyco to speed up emerge
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
alec
Apprentice
Apprentice


Joined: 19 Apr 2002
Posts: 270
Location: Here

PostPosted: Fri Mar 26, 2004 9:43 pm    Post subject: [TIP] Using psyco to speed up emerge Reply with quote

This tip was on the gentoo-user list a day or so ago; credit goes to "Gregory P. Smith".

Quote:
I was recently playing with psyco (the x86 jit-like compiler for python)
i figured i'd try out emerge with it to see what effect it had.

emerge is largely IO bound so for a first run to load the buffer cache
the effect isn't likely to be noticed either way.

however on later runs once the buffer cache was primed i found that the
use of psyco in emerge sped up "emerge search xfs" by >20% and sped up
"emerge -pU world" by ~35%. I tested using psyco 1.2 on a celery333.

I recommend adding the following to emerge:

Code:
try:
    import psyco
    psyco.full()
except ImportError:
    pass

Quote:

Anyone who has installed psyco (its an ebuild) will benefit from a faster
searching and dependancy calculations.

I also watched the memory usage during the runs as well. emerge without
psyco consumed 7.5-10mb of ram. emerge with psync consumed 9-13mb.


Emerge 'feels' faster for me; one or two others posted similar results on the list. No one has posted difficulties.

Edit: Please see the post two or three down about where to put this in '/usr/bin/emerge'. This will only speed up the program 'emerge' itself, not compiles. You'll need to do this again if you ever upgrade portage. Additionally, it isn't quite clear in his message, but you need to 'emerge psyco' to use this.


Last edited by alec on Fri Mar 26, 2004 10:02 pm; edited 1 time in total
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Fri Mar 26, 2004 9:54 pm    Post subject: Reply with quote

perhaps a silly question - but where would i add that bit?

to the end of /usr/bin/emerge?
_________________
proud to be a scout and a chronic penguin hugger
Legion of Lore - site
Back to top
View user's profile Send private message
ed0n
l33t
l33t


Joined: 23 Apr 2003
Posts: 638
Location: Prishtine/Kosove

PostPosted: Fri Mar 26, 2004 9:57 pm    Post subject: Reply with quote

neenee wrote:
perhaps a silly question - but where would i add that bit?

to the end of /usr/bin/emerge?


I wanted to ask that too.
Back to top
View user's profile Send private message
alec
Apprentice
Apprentice


Joined: 19 Apr 2002
Posts: 270
Location: Here

PostPosted: Fri Mar 26, 2004 9:57 pm    Post subject: Reply with quote

I don't know the first thing about python, but I added it right after the initial import, so my /usr/bin/emerge looks like this (from the top):

Code:
#!/usr/bin/python -O
# Copyright 1999-2003 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2
# $Header: /home/cvsroot/gentoo-src/portage/bin/emerge,v 1.264 2004/02/08 21:24:15 nakano Exp $
                                                                                                                               
import os,sys
os.environ["PORTAGE_CALLER"]="emerge"
sys.path = ["/usr/lib/portage/pym"]+sys.path
                                                                                                                               
try:
        import psyco
        psyco.full()
except ImportError:
        pass
                                                                                                                               
import emergehelp,xpak,string,re,commands,time,shutil,traceback,atexit,signal,socket,types
from stat import *
from output import *
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Fri Mar 26, 2004 10:10 pm    Post subject: Reply with quote

thanks for the prompt reply :wink:
_________________
proud to be a scout and a chronic penguin hugger
Legion of Lore - site
Back to top
View user's profile Send private message
cbr
Apprentice
Apprentice


Joined: 05 Jan 2004
Posts: 285
Location: Tallinn/Rakvere, Estonia

PostPosted: Sat Mar 27, 2004 12:04 am    Post subject: Reply with quote

OMG 8O
This speeds up searching A LOT!
Thanks for the tip :wink:
Back to top
View user's profile Send private message
discomfitor
l33t
l33t


Joined: 21 Feb 2003
Posts: 927
Location: None

PostPosted: Sat Mar 27, 2004 12:20 am    Post subject: Reply with quote

WOW. That's amazing! It fails horribly. Segfaults everywhere.
_________________
There is no substitute for experience.
Imperfection indicates a lack of effort.
Back to top
View user's profile Send private message
alec
Apprentice
Apprentice


Joined: 19 Apr 2002
Posts: 270
Location: Here

PostPosted: Sat Mar 27, 2004 2:14 am    Post subject: Reply with quote

Well, I suppose YMMV :)

I've been using it for a day on an ~x86 and a x86 system, and have had no problems.
Back to top
View user's profile Send private message
snakattak3
Guru
Guru


Joined: 11 Dec 2002
Posts: 468
Location: Seattle

PostPosted: Sat Mar 27, 2004 4:58 am    Post subject: Reply with quote

WOW. Searching is almost instantaneous! Even "emerge -p depclean" is much faster. It used to take almost a full minute sometimes. That was one thing that always bugged me about portage, slow on searching.
We should have a USE="psyco" flag, that way it can patch portage (/usr/bin/emerge) everytime you upgrade or install it. That wouldn't be too hard would it?
_________________
Ban Reality TV!
Adopt an Unanswered Post
Back to top
View user's profile Send private message
LAsk
n00b
n00b


Joined: 02 Dec 2003
Posts: 32
Location: Avesta, Sweden

PostPosted: Sat Mar 27, 2004 10:16 am    Post subject: Reply with quote

It fails for me too :(

These are the errors when I do a emerge world -p

Code:
/usr/lib/portage/pym/portage.py:2952: warning: eval()/execfile() cannot see the locals in functions bound by Psyco; consider using eval() in its two- or three-arguments form
  def match2(self,mydep,mykey,mylist):
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2176, in ?
    if not mydepgraph.xcreate(myaction):
  File "/usr/bin/emerge", line 1042, in xcreate
    if not self.create(myk):
  File "/usr/bin/emerge", line 755, in create
    if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
  File "/usr/bin/emerge", line 945, in select_dep
    if not self.create(myk,myparent,myuse=binpkguseflags):
  File "/usr/bin/emerge", line 755, in create
    if not self.select_dep("/",mydep["/"],myparent=mp,myuse=myuse):
  File "/usr/bin/emerge", line 877, in select_dep
    myeb=portage.portdb.xmatch("bestmatch-visible",x)
  File "/usr/lib/portage/pym/portage.py", line 4065, in xmatch
    myval=best(self.xmatch("match-visible",None,mydep,mykey))
  File "/usr/lib/portage/pym/portage.py", line 4076, in xmatch
    myval=self.match2(mydep,mykey,self.xmatch("list-visible",None,mydep,mykey))
  File "/usr/lib/portage/pym/portage.py", line 3017, in match2
    if eval("pkgcmp(cp_x[1:],cp_key[1:])"+cmpstr+"0"):
  File "<string>", line 0, in ?
NameError: name 'cp_x' is not defined


Anyone who knows what it is that might be wrong?
_________________
Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgement.
- Tolkien
Back to top
View user's profile Send private message
AlterEgo
Veteran
Veteran


Joined: 25 Apr 2002
Posts: 1619

PostPosted: Sat Mar 27, 2004 10:40 am    Post subject: Reply with quote

Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing :roll:
Back to top
View user's profile Send private message
ed0n
l33t
l33t


Joined: 23 Apr 2003
Posts: 638
Location: Prishtine/Kosove

PostPosted: Sat Mar 27, 2004 1:19 pm    Post subject: Reply with quote

AlterEgo wrote:
Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing :roll:

I got that result too.
Back to top
View user's profile Send private message
Vagabond
Apprentice
Apprentice


Joined: 19 Jan 2003
Posts: 192

PostPosted: Sat Mar 27, 2004 2:25 pm    Post subject: Reply with quote

you guys must have slow computers, I get 10 seconds for vanilla emerge -pu world and 8 seconds with psyco. And my portage tree is even mounted over NFS :P

Vag
Back to top
View user's profile Send private message
øxygen
Apprentice
Apprentice


Joined: 09 Mar 2004
Posts: 236
Location: Bergheim, Germany

PostPosted: Sat Mar 27, 2004 2:50 pm    Post subject: Reply with quote

or you have a small world file... btw: works
Back to top
View user's profile Send private message
kronon
Apprentice
Apprentice


Joined: 11 Aug 2003
Posts: 212
Location: NL

PostPosted: Sat Mar 27, 2004 4:19 pm    Post subject: Reply with quote

some notes to get it working.

1).you have to emerge psyco first.
2).python is to damn sensetive

this code will work


try:
import psyco
psyco.full()
except ImportError:
pass

NOTE: only difference is the space before except, this dit the trick for me.
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Sat Mar 27, 2004 10:14 pm    Post subject: Reply with quote

consider posting that bit between code tags.

unless it's meant to have no space before any
of the lines.
_________________
proud to be a scout and a chronic penguin hugger
Legion of Lore - site
Back to top
View user's profile Send private message
LAsk
n00b
n00b


Joined: 02 Dec 2003
Posts: 32
Location: Avesta, Sweden

PostPosted: Sat Mar 27, 2004 10:15 pm    Post subject: Reply with quote

kronon wrote:
some notes to get it working.

1).you have to emerge psyco first.
2).python is to damn sensetive

this code will work


try:
import psyco
psyco.full()
except ImportError:
pass

NOTE: only difference is the space before except, this dit the trick for me.


Still the same problem here :(
_________________
Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgement.
- Tolkien
Back to top
View user's profile Send private message
mastergoon
Apprentice
Apprentice


Joined: 27 Jul 2003
Posts: 161
Location: Portland, OR

PostPosted: Sun Mar 28, 2004 12:45 am    Post subject: Reply with quote

make sure you use real tabs to indent in emerge
Back to top
View user's profile Send private message
PovMan
Guru
Guru


Joined: 30 Sep 2002
Posts: 375
Location: Australia

PostPosted: Sun Mar 28, 2004 3:14 am    Post subject: Reply with quote

For anyone who's having segfaults, make sure you do an 'emerge psyco' before modifying emerge
It works great for me! Why isn't this part of emerge already? (like as a use flag)
_________________
I am SPAM, hear me roar.
Before posting your own topic, try to answer at least one unanswered one.
Back to top
View user's profile Send private message
verbatim
Apprentice
Apprentice


Joined: 13 Mar 2003
Posts: 223

PostPosted: Sun Mar 28, 2004 3:42 am    Post subject: Reply with quote

Hey, works for me... getting a measureable speedup.

user time for emerge -epv world without: 33 seconds
with: 22 seconds

Cool. :)
Back to top
View user's profile Send private message
schism39401
Tux's lil' helper
Tux's lil' helper


Joined: 13 Mar 2003
Posts: 130

PostPosted: Sun Mar 28, 2004 5:12 am    Post subject: Reply with quote

Searching does seem a little faster...I use esearch usually and it's a lot faster :-)
Back to top
View user's profile Send private message
djkork
n00b
n00b


Joined: 25 May 2004
Posts: 45
Location: Madrid

PostPosted: Tue May 25, 2004 1:40 am    Post subject: Reply with quote

this does not speed up too much....

emerge -puD world

20 sec with psico
18 sec with psico
Back to top
View user's profile Send private message
gbu
n00b
n00b


Joined: 18 May 2004
Posts: 25
Location: Between UCI and UCLA

PostPosted: Tue May 25, 2004 10:14 pm    Post subject: Reply with quote

Of course you can do the same thing to esearch and eupdatedb.
Back to top
View user's profile Send private message
equilibrium
Apprentice
Apprentice


Joined: 29 Jun 2003
Posts: 213
Location: UK

PostPosted: Wed May 26, 2004 11:48 am    Post subject: Reply with quote

omg nice :)

searching seems faster :D will see how fast it is to emerge some stuff in future. Am thinking of upgrading KDE 8)
_________________
kernel 3.14.4-2-ck | i3wm | github
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Wed May 26, 2004 12:09 pm    Post subject: Reply with quote

emerge -uDpv world,

without psyco:

Code:
real    0m7.071s
user    0m5.472s
sys     0m0.631s


with psyco:

Code:
real    0m4.572s
user    0m3.420s
sys     0m0.648s


ps. i only timed the second and third time, to be sure the difference
in time was not because of the files/process being cached.
_________________
proud to be a scout and a chronic penguin hugger
Legion of Lore - site
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum