| View previous topic :: View next topic |
| Author |
Message |
Thistled
Guru
Joined: 06 Jan 2011
Posts: 433
Location: Scotland
|
 Posted: Tue Apr 10, 2012 9:29 pm Post subject: [SOLVED] PyPy and lack of memory to build |
 |
|
Apologies guys if this is not the correct forum. I will not be surprised to see it has been moved to Unsupported.
I have a 2GB system, and when I try to build the latest ~x86 pypy, I am told I do not meet the minimum 2GB requirements.
Is there a way to tweak this / my system, or even the ebuild itself so it will compile.
On a fresh boot, my system (without GDM running) sitting at console is consuming just 89Mb, so I have 1900 odd MB available, and 2GB swap.
I was hoping to leave the system compiling pypy through the night, and let it swap out to it's hearts content, but that is not going to be possible it seems.
I'm not too comfortable with "interfering" with ebuilds, and was wondering if the only solution is to "adjust" the pre-emerge checks within the ebuild. 
_________________
Whatever you do, do it properly!
Last edited by Thistled on Thu May 17, 2012 8:16 pm; edited 1 time in total |
|
| Back to top |
|
 |
wthrowe
Tux's lil' helper
Joined: 19 Aug 2009
Posts: 134
|
| Posted: Wed Apr 11, 2012 3:03 am Post subject: |
|
|
Try | Code: | | I_KNOW_WHAT_I_AM_DOING=1 emerge pypy |
Some eclasses (and maybe ebuilds) allow you to bypass dies with that. You could put it in make.conf, but I'm not sure how wide an effect it has. |
|
| Back to top |
|
|
Thistled
Guru
Joined: 06 Jan 2011
Posts: 433
Location: Scotland
|
| Posted: Wed Apr 11, 2012 12:25 pm Post subject: |
|
|
wthrowe,
You Genius!
I noticed the initial "I_KNOW_WHAT_I_AM_DOING" in the "die" of my first attempt at trying to build it, but was unaware I could use such a "flag" to force it.
It still warns me of impending failure in the ebuild, but I will give it a try when I log out and free up the majority of my RAM tonight before hitting the sack.
Thanks again man!
_________________
Whatever you do, do it properly! |
|
| Back to top |
|
|
jdhore
Developer
Joined: 13 Apr 2007
Posts: 105
|
| Posted: Wed Apr 11, 2012 5:19 pm Post subject: |
|
|
| Thistled wrote: | wthrowe,
You Genius!
I noticed the initial "I_KNOW_WHAT_I_AM_DOING" in the "die" of my first attempt at trying to build it, but was unaware I could use such a "flag" to force it.
It still warns me of impending failure in the ebuild, but I will give it a try when I log out and free up the majority of my RAM tonight before hitting the sack.
Thanks again man! |
Just as a FYI, on my single-core 2.2GhZ Celeron machine with 2GB RAM (and only like 90MB in use by other stuff), building pypy 1.8 took a bit over 3 hours. |
|
| Back to top |
|
|
Thistled
Guru
Joined: 06 Jan 2011
Posts: 433
Location: Scotland
|
| Posted: Sat May 05, 2012 12:27 pm Post subject: |
|
|
So I finally took the jump, and rebooted my system in order to have a minimal usage of RAM and built pypy through the night.
It built fine. No shortage of RAM but, I have now rebuilt portage minus the python use flags.
Any ideas on how to get it to build using the pypy1_8 use flag?
I have set it in /etc/portage/package.use but it still won't build it using the flag.
gives me
| Code: | * sys-apps/portage
Available versions: [M]2.1.6.7 2.1.6.13 2.1.10.49 (~)2.1.10.57 ~*2.2.0_alpha89 ~*2.2.0_alpha101 **9999
IUSE (all versions): build doc epydoc +ipc (+)less linguas_pl pypy1_8 python2 python3 selinux xattr
Installed versions: Version: 2.1.10.57
Date: 13:17:37 05/05/12
USE: ipc xattr -build -doc -epydoc -linguas_pl -pypy1_8 -python2 -python3 -selinux
Best versions/slot: (~)2.1.10.57
Homepage: http://www.gentoo.org/proj/en/portage/index.xml
Find open bugs: http://bugs.gentoo.org/buglist.cgi?quicksearch=sys-apps%2Fportage
Description: Portage is the package management and distribution system for Gentoo
License: GPL-2
|
Do I have to add -pypy1_8 to /etc/portage/packages.use.unmask ??
Many thanks in advance.
_________________
Whatever you do, do it properly! |
|
| Back to top |
|
|
tranquilcool
Veteran
Joined: 25 Mar 2005
Posts: 1035
|
| Posted: Sat May 05, 2012 6:43 pm Post subject: |
|
|
Ummask it in /etc/portage/profile
Ok create a file use_unmask in /etc/portage/profile.
Then in the file write -pypy1_8
Then build portage.
_________________
this is a strange strange world.
Last edited by tranquilcool on Sat May 05, 2012 10:08 pm; edited 1 time in total |
|
| Back to top |
|
|
jdhore
Developer
Joined: 13 Apr 2007
Posts: 105
|
| Posted: Sat May 05, 2012 7:03 pm Post subject: |
|
|
I feel i'll point out that...I did this when pypy support was first added to portage and...It really didn't seem any faster between running time against it and it didn't feel faster either.
If you really want portage to perform better, i'd suggest putting /usr/portage on a SquashFS (there are several initscripts to help you with this) or a tmpfs or switch to a SSD. The longest component of dependency calculation is that portage has to go through MANY small flat files which is not fast, even on filesystems optimized for those sorts of files. |
|
| Back to top |
|
|
tranquilcool
Veteran
Joined: 25 Mar 2005
Posts: 1035
|
| Posted: Sat May 05, 2012 10:14 pm Post subject: |
|
|
When i tried squashfs, i had corruptions too often for my taste.
Personally, i like to experiment with something new. Right now, it's
portage with pypy.
_________________
this is a strange strange world. |
|
| Back to top |
|
|
Thistled
Guru
Joined: 06 Jan 2011
Posts: 433
Location: Scotland
|
| Posted: Sun May 06, 2012 3:18 pm Post subject: |
|
|
tranquilcool, many thanks!
I was kind of on the right track, I had put it in /etc/portage/package.use.unmask
No wonder it didn't work.
It's rebuilt now with pypy, now lets see if there is a slight improvement. (Not that I had a problem with compile times anyway)
Cheers.
_________________
Whatever you do, do it properly! |
|
| Back to top |
|
|
jdhore
Developer
Joined: 13 Apr 2007
Posts: 105
|
| Posted: Sun May 06, 2012 5:43 pm Post subject: |
|
|
| Thistled wrote: | tranquilcool, many thanks!
I was kind of on the right track, I had put it in /etc/portage/package.use.unmask
No wonder it didn't work.
It's rebuilt now with pypy, now lets see if there is a slight improvement. (Not that I had a problem with compile times anyway)
Cheers. |
Well PyPy won't really help at all with compile times as basically all the time taken there is taken in running configure and make and make install. VERY little python code is being run to the point that it can't really help. |
|
| Back to top |
|
|
Thistled
Guru
Joined: 06 Jan 2011
Posts: 433
Location: Scotland
|
| Posted: Sun May 06, 2012 6:22 pm Post subject: |
|
|
jdhore, thanks for that. I just ran a few emerges and was thinking just that. There isn't really any noticeable difference.
Did I not read somewhere on these forums that building portage with Python-3 is more beneficial?
I.E. Portage works much quicker under Python-3 rather than Python-2?
_________________
Whatever you do, do it properly! |
|
| Back to top |
|
|
jdhore
Developer
Joined: 13 Apr 2007
Posts: 105
|
| Posted: Mon May 07, 2012 12:37 am Post subject: |
|
|
| Thistled wrote: | jdhore, thanks for that. I just ran a few emerges and was thinking just that. There isn't really any noticeable difference.
Did I not read somewhere on these forums that building portage with Python-3 is more beneficial?
I.E. Portage works much quicker under Python-3 rather than Python-2? |
I have heard that and i'm using python3 for my portage. It doesn't seem much faster, and if the old buildbot is anything to go by, it might be slightly slower than Python 2.7 (Python 3.3 might be faster and better because it's dropped its memory usage by a lot, but that's not out yet, not in portage yet, may eat your children, etc).
As i mentioned in my earlier post, while none of them are great ideas, your 3 best options are tmpfs, squashfs or SSD. I played with the SquashFS solution on my server and it *DOES* make a huge performance difference (it was probably nearly twice as fast as before), I just didn't like it because it has to rebuild the squashfs on every sync and I sync quite often so that was kind of annoying. |
|
| Back to top |
|
|
|
Portage & Programming
