Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[py] running out of RAM?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Oct 06, 2012 1:33 pm    Post subject: [py] running out of RAM? Reply with quote

So I wanted to play a little with a (rather) simple statistics problem going through my mind, but python or the system is crapping out on me.

Not even started to write the whole procedure, just this simple loop
Code:
for i in range(0, j):
  pass


Now, for j=10**10, I get
Code:
Traceback (most recent call last):
  File "./test.py", line 10, in <module>
    for i in range(0, j):
MemoryError


For j=10**9, the process runs for a while and then gets killed by zsh/something, seeing in `top` a raise of ~7GB RAM before being killed of. For j<=10**8 everything works fine.

I need to run at least 10^9 samples, 10^10 would be better, but why is it crapping out though doing practically nothing?

Edit, in comparison, the equivalent in ruby
Code:
a = 10**10

1.upto(a) { |k|
    if k % 10**6 == 0
        puts k
    end
}
works just fine even without claiming a lot of RAM(<=1.5GB).
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.
Back to top
View user's profile Send private message
dol-sen
Retired Dev
Retired Dev


Joined: 30 Jun 2002
Posts: 2805
Location: Richmond, BC, Canada

PostPosted: Sat Oct 06, 2012 2:53 pm    Post subject: Reply with quote

Definitely sounds like a bug in python. I ran your test on my 4G ram system and 10**8 stuffed my whole memory and pushed into swap.

But a slight variation on that and with a new python session (so memory usage was reset) runs without increasing python's starting memory. It is running 10**10 for me now... erh, still ;)

Code:
>>> def t2(j):
...     i=0
...     while i<=j:
...             i+=1
...
>>> t2(10**10)


ok, 10**10 ran to completion on my system without an increase in python's starting memory. There is obviously something wrong with either python's range() or for loop. But a while loop will do the same without breaking a sweat or straining memory at all.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Oct 06, 2012 2:58 pm    Post subject: Reply with quote

Incredibly inefficient. range() is nice and elegant but doesn't work well for large ranges. At 10**10, you're asking it to create a list object with 10,000,000,001 elements. Seem like it might take up a lot of memory, doesn't it? Try an alternate implementation using traditional integers and a while construct.

Edit: Yes, exactly like above. :wink:

It's not a bug. If you look at the range() function definition, it states that it, "...generates a list...". 10**10 makes a big list.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Sat Oct 06, 2012 3:36 pm    Post subject: Reply with quote

For this use case, you should use xrange, which returns a generator.
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Oct 06, 2012 3:40 pm    Post subject: Reply with quote

Mh, why does it even create a list and not just do the 'x+(y*step)' on itself while processing? Isn't enumerate() there for the list-stuff?

I get what you're saying, but isn't this an overly complicated non python'ish way? While, according to your description, still somewhat logical, it's a trap for newbies, imho. I could have thought doing it the while-way, but after that simple aproach failed, my first guess was that something is wrong with the machine/python.

The Ruby version works as expected, though I didn't look how it's done internally. Haven't tried Perl, but I'm pretty sure it'll work.

BTW, I should have made this threaded somehow, while 10M iterations/5 secs on a single core is ok, still takes looong :p

Edit, @Hu, thanks.
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.


Last edited by avx on Sat Oct 06, 2012 3:48 pm; edited 1 time in total
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Oct 06, 2012 3:45 pm    Post subject: Reply with quote

Hu wrote:
For this use case, you should use xrange, which returns a generator.
Bah. Wrong fix for the (language structure) problem, desperately avoiding traditional iteration concepts. Of course, in the concept of the language structure, you're correct; I'm a Python noob and didn't know that. :wink:

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21431

PostPosted: Sat Oct 06, 2012 5:44 pm    Post subject: Reply with quote

avx: per pydoc enumerate, it appears to generate a list of pairs of (index, iterator[index]), which is not useful here. I suspect that range was created first because it was easy, while xrange requires language support for generators via the yield directive. I agree that this implementation of range is not friendly to newcomers, especially since pydoc range does not mention the existence of xrange. The existing implementation of range can be useful if you want to save the result into a temporary and do non-linear strides across the list, especially if you need to move forward sometimes and backward other times.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum