Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Python-2/3 vs pypy (not recommendable) as "emerge" backends
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Mon Apr 02, 2012 1:02 pm    Post subject: Python-2/3 vs pypy (not recommendable) as "emerge" Reply with quote

In this thread, the performances of python-3 (and 2) and pypy are compared to run "emerge".

Conclusion: Stick to python-3, although it seems to vary greatly among environments. There doesn't seem to exist any special reason to use pypy(-bin), at least for pypy-1.9 and 2.0, although it's often said that pypy is far faster than Cpython.

Edit history
* Apr 2012: portage can use pypy as a backend.
* 1 Aug 2013: pypy-bin is in portage tree.
* 18 Aug 2013: performance comparison data have accumulated.

How to use pypy(-bin)-1.9 as the portage backend:
0. (pypy(-bin):1.9 is now masked. pypy:2.0 is available, but pypy-bin-2.0.2 can't run portage.)

1. Add
sys-apps/portage -pypy1_9
to /etc/portage/profile/package.use.mask (the preceding hyphen in "-pypy1_9" is correct here.) to unmask a USE flag.

2. Add
sys-apps/portage pypy1_9
to /etc/portage/package.use.

3. Add
dev-python/pypy-bin:1.9
virtual/pypy:1.9
to /etc/portage/package.keywords

4. Re-emerge sys-apps/portage, and you're done!

Notes:

pypy vs pypy-bin
Building of pypy takes long; according to my slack observation, it seems some 13 times of building a kernel. But you can avoid building by emerging dev-python/pypy-bin:1.9, the binary package of pypy.

pypy-bin: 1.9 vs 2.0
As for the pypy-bin, you can't install sys-apps/portage with =dev-python/pypy-bin-2.0.2, but =dev-python/pypy-bin-1.9 works. With 2.0.2, the sanity check fails:
build log wrote:
>>> Merging sys-apps/portage-2.1.12.2 to /
* Running preinst sanity tests...
testBashSyntax (portage.tests.lint.test_bash_syntax.BashSyntaxTestCase) ... ok
testCompileModules (portage.tests.lint.test_compile_modules.CompileModulesTestCa se) ... ok
testImportModules (portage.tests.lint.test_import_modules.ImportModulesTestCase) ... FAIL


I only tested pypy-bin-2.0.2. From what we gather here, a home-built pypy-2.0.2 with appropriate USE flags seems ok for portage.

Unmasking a USE flag
Currently, the USE flag pypy1_9 is masked. You can unmask it by preparing /etc/portage/profile/package.use.mask. User "cgw" said below to use /etc/portage/make.profile/..., but /etc/portage/make.profile is a symlink to somewhere under /usr/portage/profile, and you shouldn't use it. /etc/portage/profile is for your personal configuration.

If you also want to use pypy(-bin) for packages other than sys-apps/portage, then you have to unmask another USE flag, "python_targets_pypy1_9".

If you screwed up things, you can install the portage binary from the tinderbox. default-linux/<your arch> is a moderate choice.

Thanks all.
_________________
Easy Shift / Ctrl / AltGr ... hack; save your pinkies, type without drudge: topic 865313


Last edited by teika on Sun Aug 18, 2013 11:25 am; edited 4 times in total
Back to top
View user's profile Send private message
cgw
n00b
n00b


Joined: 04 Oct 2012
Posts: 2

PostPosted: Thu Oct 04, 2012 4:40 pm    Post subject: Is anyone using pypy for portage? Reply with quote

Is anyone using pypy for portage? I tried "USE=-python2 -python3 pypy1_9" and /usr/bin/emerge still runs with cpython2.7, not pypy. I created (possibly prematurely?) a bug report for this, containing more detail.

https://bugs.gentoo.org/show_bug.cgi?id=437198
Back to top
View user's profile Send private message
cgw
n00b
n00b


Joined: 04 Oct 2012
Posts: 2

PostPosted: Thu Oct 04, 2012 5:56 pm    Post subject: fixed! Reply with quote

Turns out the USE flag was masked (in /usr/portage/profiles/package.use.mask). Adding
a file /etc/portage/profile/package.use.mask/portage.use.mask containing the line
sys-apps/portage -pypy1_9
is sufficient to unmask the pypy1_9 USE flag.

(Clearly, I didn't know about masked USE flags, with Gentoo there's always something new to learn!)
Back to top
View user's profile Send private message
one_and_only
Apprentice
Apprentice


Joined: 13 May 2007
Posts: 249
Location: PL/Krakow

PostPosted: Thu Nov 01, 2012 12:09 pm    Post subject: Reply with quote

anybody using/hacking pypy backend?

edit: quick benchmark of emerge -auND @world for quite not up-to-date system (time of resolving update graph, not the whole emerging):
Code:

pypy:
emerge --update --newuse --keep-going --deep --tree --quiet-build=y  -a @worl  168.28s user 5.95s system 99% cpu 2:55.16 total

python3:
emerge --update --newuse --keep-going --deep --tree --quiet-build=y  -a @worl  271.40s user 6.33s system 99% cpu 4:38.64 total

python2:
emerge --update --newuse --keep-going --deep --tree --quiet-build=y  -a @worl  323.13s user 6.50s system 99% cpu 5:30.59 total
Back to top
View user's profile Send private message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Thu Aug 01, 2013 5:24 am    Post subject: portage w/ pypy-bin howto Reply with quote

Now pypy-bin is in the portage tree. I've posted a howto, by rewriting the first post of this thread. Thanks cgw for the howto of unmasking USE flags.
Back to top
View user's profile Send private message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Fri Aug 09, 2013 2:11 am    Post subject: Reply with quote

Yesterday, pypy-bin:1.9 was masked. =P
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4123

PostPosted: Fri Aug 09, 2013 7:37 am    Post subject: Reply with quote

Building pypy requires the insane amount of 2GB RAM. Before I make my system slower or even useless: Can anybody tell me how much RAM is required with pypy-2.0 for running portage, e.g.
Code:
/usr/bin/time -f '%M kB'  emerge -peD @world
with perhaps 1500 installed packages?
Back to top
View user's profile Send private message
Naib
Advocate
Advocate


Joined: 21 May 2004
Posts: 4119
Location: Removed by Neddy

PostPosted: Fri Aug 09, 2013 7:29 pm    Post subject: Reply with quote

shouldn't this be in doc's and tips?
_________________
A free press is the unsleeping guardian of every other right that free men prize; it is the most dangerous foe of tyranny. Where men have the habit of liberty, the Press will continue to be the vigilant guardian of the rights of the ordinary citizen.
Back to top
View user's profile Send private message
Civil
n00b
n00b


Joined: 24 Feb 2006
Posts: 15
Location: Moscow

PostPosted: Wed Aug 14, 2013 6:16 pm    Post subject: Reply with quote

1413 packages. Portage 2.2.0:
pypy2_0:
Code:
77.91user 1.91system 1:19.79elapsed 100%CPU (0avgtext+0avgdata 3131264maxresident)k
0inputs+48outputs (0major+451657minor)pagefaults 0swap


python3:
Code:
49.80user 1.51system 0:51.25elapsed 100%CPU (0avgtext+0avgdata 1467344maxresident)k
0inputs+48outputs (0major+368221minor)pagefaults 0swaps


python2.7:
Code:
57.23user 1.68system 0:58.87elapsed 100%CPU (0avgtext+0avgdata 1416000maxresident)k
0inputs+48outputs (0major+364283minor)pagefaults 0swaps


Conclusion: Twice ram, 30% slower.
HW: Core i7-3517U (2 cores, 1.9GHz), 4GB Ram, SSD.
_________________
‘Who controls the past controls the future: who controls the present controls the past.’
George Orwell ‘1984’
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4123

PostPosted: Wed Aug 14, 2013 7:54 pm    Post subject: Reply with quote

Civil wrote:
Conclusion: Twice ram, 30% slower.
HW: Core i7-3517U (2 cores, 1.9GHz), 4GB Ram, SSD.

Yes, it seems with <4GB one better should not attempt to use it. Thanks.
Back to top
View user's profile Send private message
derlafff
n00b
n00b


Joined: 15 Aug 2013
Posts: 1

PostPosted: Thu Aug 15, 2013 9:49 am    Post subject: Reply with quote

I tested performance of -peD with 780 packages installed:

Code:
python3.2:
real   0m15.112s
user   0m14.817s
sys   0m0.219s
pypy2.0:
real   0m31.482s
user   0m30.619s
sys   0m0.352s


Intel Core i5-3230M, 8Gb RAM, HDD

pypy is 2 times slower, python2.7 is a little bit slower than python 3.2
Back to top
View user's profile Send private message
teika
n00b
n00b


Joined: 19 Feb 2011
Posts: 68
Location: japan

PostPosted: Sun Aug 18, 2013 11:00 am    Post subject: Reply with quote

Hmm, it seems to greatly depend. Conclusion for me is: they scarcely differ. The output of "/usr/bin/time emerge -peD":

Code:
pypy-bin-1.9:

119.26user 2.99system 2:58.14elapsed 68%CPU (0avgtext+0avgdata 611440maxresident)k
0inputs+0outputs (1major+46678minor)pagefaults 0swaps

(drop cache)
python-3.2:

119.56user 3.22system 3:02.41elapsed 67%CPU (0avgtext+0avgdata 611200maxresident)k
0inputs+0outputs (51major+46466minor)pagefaults 0swaps


Single core amd64, 615 packages. (Very slow machine. ;-)

Maybe pypy-2.0 is significanly slower and uses more RAM than pypy-1.9 ?
Back to top
View user's profile Send private message
Civil
n00b
n00b


Joined: 24 Feb 2006
Posts: 15
Location: Moscow

PostPosted: Tue Aug 20, 2013 12:21 am    Post subject: Reply with quote

I've tested on desktop (Core i7-4770@4.4GHz, 16GB Ram, HDD):
1510 pkgs

python2.7:
Code:
real    1m20.119s
user    1m19.284s
sys     0m0.825s
78.53user 0.82system 1:19.36elapsed 99%CPU (0avgtext+0avgdata 1817632maxresident)k
0inputs+0outputs (0major+127789minor)pagefaults 0swaps



python3:
Code:
real    1m6.659s
user    1m5.835s
sys     0m0.813s
65.79user 0.81system 1:06.63elapsed 99%CPU (0avgtext+0avgdata 1876432maxresident)k
200inputs+0outputs (1major+130936minor)pagefaults 0swaps


pypy2_0:
Code:
real    1m1.223s
user    1m0.134s
sys     0m1.078s
59.16user 1.02system 1:00.19elapsed 99%CPU (0avgtext+0avgdata 4344896maxresident)k
0inputs+192outputs (0major+303639minor)pagefaults 0swaps

_________________
‘Who controls the past controls the future: who controls the present controls the past.’
George Orwell ‘1984’
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum