[TIP] Using psyco to speed up emerge

Message

jpc82 · Post by **jpc82** » Wed May 26, 2004 12:45 pm

emerge -pUD world went from 10sec to 8sec
emerge -p depclean went from 7sec to 5 sec

papabean · Post by **papabean** » Thu Jun 03, 2004 10:47 am

What effect, if any, does this have on emerge sync?

MK · Post by MK » Thu Jun 03, 2004 11:56 am

Hm, tried it and it does improve speed a little

emerge -up world from 5.5s to 4.2s
emerge -s psyco from 1.8s to 1.3s
emerge -S psyco from 36s to 28s

Helps a little so I'll keep it

alec · Post by **alec** » Thu Jun 03, 2004 4:26 pm

papabean wrote:What effect, if any, does this have on emerge sync?

Probably none. This only speeds up the execution of the python code in /usr/bin/emerge - 'emerge sync' farms everything out to rsync, so you will probably see no improvement in that department. I find it speeds up dependency calculations and searches.

papabean · Post by **papabean** » Thu Jun 03, 2004 4:39 pm

alec wrote:
papabean wrote:What effect, if any, does this have on emerge sync?
Probably none. This only speeds up the execution of the python code in /usr/bin/emerge - 'emerge sync' farms everything out to rsync, so you will probably see no improvement in that department. I find it speeds up dependency calculations and searches.

Thanks. I have noticed that searches seem much faster.

Pink · Post by **Pink** » Thu Jun 03, 2004 4:40 pm

I haven't benchmarked it, but it does seem to make a difference here - even if it is only a few seconds, it's got to be worth it.

knarf · Post by **knarf** » Sat Jun 05, 2004 10:25 pm

With :

Code: Select all

real    0m28.690s
user    0m21.925s
sys     0m2.540s

Without :

Code: Select all

real    0m26.932s
user    0m24.073s
sys     0m2.663s

Species · Post by **Species** » Mon Jun 07, 2004 3:28 pm

same speedup effect as described on my laptop

without psyco:

Code: Select all

# emerge -pvDe world
real 0m17.276s
user 0m13.736s
sys 0m3.488s

with psyco:

Code: Select all

# emerge -pvDe world
real 0m12.917s
user 0m9.789s
sys 0m3.002s

Gil-galad55 · Post by **Gil-galad55** » Wed Jun 16, 2004 1:57 pm

Since this is an I/O bound process, most likely those people with significant time differences have either small world files (less I/O required) or slower processors, where the more efficient code helps out. As the world file increases, I can see the two times converging as I/O dominates.

guard0 · Post by **guard0** » Wed Jun 16, 2004 2:59 pm

i doubt it would be of much worth as a USE flag, but as a FEATURES flag (where you put distcc, ccache, sandbox, etc) since it would primarily be used for portage utilities...

aridhol · Post by **aridhol** » Thu Jun 17, 2004 4:49 pm

For you speedfreaks it might be interested to know you can use mySQL to do your searches:

http://forums.gentoo.org/viewtopic.php?t=175461

madmango · Post by **madmango** » Thu Jun 17, 2004 5:04 pm

time emerge -peD world

without: 5.997 sec
with: 5.977 sec

wOw! Amazing .2 second decrease!

polle · Post by **polle** » Thu Jun 17, 2004 6:48 pm

I get this error:

File "/usr/bin/emerge", line 12
except ImportError:
^
SyntaxError: invalid syntax

I tried to copy and paste from the above but still get that error
(used nano to edit the file /usr/bin/emerge and tried with and without a space before except)

fca · Post by **fca** » Fri Jul 02, 2004 10:50 am

unfortunately this doesn't work on a Pax protected system, The psyco compiler gets killed then.

Code: Select all

Jul  2 12:41:29 [kernel] PAX: execution attempt in: <anonymous mapping>, 17d6a000-17e1e000 17d6a000

c0balt · Post by **c0balt** » Thu Aug 05, 2004 3:22 pm

Hi,
couldnt this be added to portage officially?
afaik the code doesnt brake anything if psyco isnt installed.
everytime theres a new portage package one has to add the lines manually, though thats not that often its annoying and i tend to forget

blu3bird · Post by **blu3bird** » Sat Aug 07, 2004 9:35 am

emere -up world:

Code: Select all

without psyco:
7 Secunds
with psyco:
5 Seconds

emere -s gentoo:

Code: Select all

without psyco:
3 Secunds
with psyco:
3 Seconds

not as fast as it should be, but okay

thinko · Post by **thinko** » Tue Sep 14, 2004 7:22 am

I think you guys haven't correctly configured psyco.. when I use it with emerge, I see a significant benefit every time.

ex: emerge -pDuv world
Without Psyco:

Code: Select all

real    0m34.392s
user    0m23.512s
sys     0m3.056s

With Psyco:

Code: Select all

real    0m21.905s
user    0m16.532s
sys     0m2.709s

or.. emerge -pe world
Without Psyco:

Code: Select all

real    0m34.925s
user    0m23.368s
sys     0m3.142s

With Psyco:

Code: Select all

real    0m24.636s
user    0m17.210s
sys     0m2.844s

Funny, I think it's well worth it.

Post by **Pythonhead** » Tue Sep 14, 2004 10:11 pm

All these tests are pretty meaningless. I notice everyone is timing the psyco version second. All this stuff is getting cached, so whenever you run a second test, it should be faster, psyco or not:

I boot up my system and run emerge -ep world:

real 0m51.199s
user 0m15.540s
sys 0m1.467s

Then run it again immediately:

real 0m25.222s
user 0m15.376s
sys 0m1.192s

(neither test using psyco)

There was already a bug filed asking to put psyco in portage (where requests go, not these forums, by the way) and it was declined:

http://bugs.gentoo.org/show_bug.cgi?id=45793

Also, psyco also uses an enormous amount of memory, so wouldn't be great for everyone.

psyco may help a tiny bit for searching portage, but its almost all disk i/o that is the bottleneck. Thats why people aren't getting much if any speedups with the mysql portage backend (last I checked).

neysx · Post by **neysx** » Tue Sep 14, 2004 10:57 pm

Psyco can improve performances significantly depending on CPU/IO/World file.

On a Via C3-800
#1 w/psyco:
real 0m30.547s
user 0m12.717s
sys 0m1.980s

#2 w/ psyco again (cached data):
real 0m15.492s
user 0m12.352s
sys 0m1.694s

#3 wo/psyco (data still in cache):
real 0m21.210s
user 0m17.691s
sys 0m1.541s

Another run wo/psyco yields the same values.

Same test on an old P100 (2x with psyco, 1x without)
real 1m54.831s
user 1m38.690s
sys 0m7.410s

real 1m45.879s
user 1m38.610s
sys 0m6.620s

real 2m25.927s
user 2m18.590s
sys 0m6.510s

On my 2xMp2400+ (huge world file)
with psyco
real 1m25.650s
user 1m4.695s
sys 0m10.188s

without
real 1m31.772s
user 1m12.129s
sys 0m10.334s

Generally speaking, using psyco does not make much sense on fast modern machines but it has its uses on old equipment.

fbd · Post by **fbd** » Sat Oct 09, 2004 12:19 pm

AlterEgo wrote:Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing

Try using emerge -s somename

You will see the diffrence

v0m1t · Post by **v0m1t** » Sat Oct 09, 2004 1:22 pm

emerge -s xmms with psyco
real 0m11.637s
user 0m4.909s
sys 0m0.480s

emerge -s xmms without psyco
real 0m8.575s
user 0m5.106s
sys 0m0.454s

Both tests with cached results.

redshift · Post by **redshift** » Tue Oct 12, 2004 2:46 pm

With portage 2.0.51_rc9, this seems to give me constant segfaults. Before emerging rc9 today (from rc7) I never had problems, and psyco did give a little boost.

Trevoke · Post by **Trevoke** » Tue Oct 12, 2004 2:53 pm

Same issue here. New portage no like psyco.

Jtb · Post by **Jtb** » Thu Oct 14, 2004 10:00 am

Just tried it on a fast machine (AMD64 3000+ with 1GB but 32bit Gentoo):

time emerge search psyco
with:
real 0m9.545s
user 0m0.778s
sys 0m0.191s

without:
real 0m7.229s
user 0m0.946s
sys 0m0.165s

time emerge -S psyco
with:
real 1m0.649s
user 0m16.814s
sys 0m3.649s

without:
real 1m11.777s
user 0m29.205s
sys 0m3.637s

time emerge -ep world
without:
real 0m44.441s
user 0m13.307s
sys 0m1.066s

with:
real 0m16.807s
user 0m8.227s
sys 0m0.968s

Reboots between with and without:
emerge search psyco, emerge -S psyco [reboot] emerge search psyco, emerge -S psyco [reboot emerge -ep world [reboot] emerge -ep world

Viha · Post by **Viha** » Thu Oct 14, 2004 12:06 pm

Segfaulted here on emerge clean (portage 2.0.51_rc9).

[TIP] Using psyco to speed up emerge

Emerge runtimes with psyco enabled.

:)