Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
really LARGE malloc
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
pilla
Administrator
Administrator


Joined: 07 Aug 2002
Posts: 7225
Location: Pelotas, BR

PostPosted: Tue Sep 17, 2002 5:11 pm    Post subject: really LARGE malloc Reply with quote

hullo,

I'm trying to allocate 2 GB of memory in a single malloc call in C. But it doesn't work, saying that I don't have enough memory.

I'm trying to find what is the real problem here (it is not memory, because I have 4 GB). Maybe it is a limit on the maximum a process can get. I tried to find something in the rlimit functions, but wasn't able to.

The kernel was compiled for 4 GB and it sees the memory. I understand that 512 MB is allocated to the kernel, as it doesn't appear in the memory summaries.



Thanks in advance.
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Tue Sep 17, 2002 6:05 pm    Post subject: Reply with quote

If you're using x86 hardware, IIRC more than 2GB is probably not addressable as a single block (i.e. you're pushing architectual limitations). I'm not sure if that's the exact limit... Anyhow, I would see if you could break it into smaller pieces (i.e. have a few 1 gig mallocs) and see if that works... It might be impossible with current processors/kernels to address more than that in a single process.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
pilla
Administrator
Administrator


Joined: 07 Aug 2002
Posts: 7225
Location: Pelotas, BR

PostPosted: Tue Sep 17, 2002 6:10 pm    Post subject: Reply with quote

Yes, this was my first feelling, I couldn't remember the exacts limits of x86 hardware. I'm doing some research elsewhere. The next thing I'll try is a 64 GB memory kernel, maybe with a different memory model I'll be able to access all the memory (or at least 3 GB).

EDIT: seems that there is some problem with glibc/malloc. I can only allocate ~ 2 GB.

phong wrote:
If you're using x86 hardware, IIRC more than 2GB is probably not addressable as a single block (i.e. you're pushing architectual limitations). I'm not sure if that's the exact limit... Anyhow, I would see if you could break it into smaller pieces (i.e. have a few 1 gig mallocs) and see if that works... It might be impossible with current processors/kernels to address more than that in a single process.
Back to top
View user's profile Send private message
pilla
Administrator
Administrator


Joined: 07 Aug 2002
Posts: 7225
Location: Pelotas, BR

PostPosted: Tue Sep 17, 2002 8:57 pm    Post subject: Reply with quote

OK, we have two limits here (according to my research)

First, Linux kernel separates 1 GB for the OS and 3 GB for user processes in order not to incur in performance loss (problems of x86 architecture).

Second, a process may have only 2 GB for it.

Please, shoot me if I'm saying something stupid/wrong.
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Sep 17, 2002 9:08 pm    Post subject: Reply with quote

I found this admittedly a bit old LK message from kernel hacker Rik van Riel, in which it claims that a process can allocate 3GB, as long as it uses mmap, if this helps.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
pilla
Administrator
Administrator


Joined: 07 Aug 2002
Posts: 7225
Location: Pelotas, BR

PostPosted: Tue Sep 17, 2002 9:20 pm    Post subject: Reply with quote

I've found this quite interesting document about Linux RAM limits.

Some interesting snippets:

* Linux 2.4.x does support greater then 4GB of physical memory (ie. RAM). It does this by supporting Intel's PAE (Physical Address Extension) features which are in all Pentium Pro and newer CPU's.

* PAE does not increase Linux's ability for *single* processes to see greater then 3GB of RAM (see below).

* With 2.4 kernels (with a large memory configuration) a single process can address up to the total amount of RAM in the machine minus 1GB (reserved for the kernel), to a maximum 3GB

As long as I could find out, newer mallocs use both mmap and brk (the later when it wants to allocate small chunks. mmap is more efficient for large chunks). There is also the problem of mapping shared libraries inside the process memory space, which reduces the memory that can be allocated even when it still has a 1 GB free 8)

I'll try to keep this thread up-to-date with my findings (if I have some 8))


rac wrote:
I found this admittedly a bit old LK message from kernel hacker Rik van Riel, in which it claims that a process can allocate 3GB, as long as it uses mmap, if this helps.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum