really LARGE malloc

pilla · Posted: Tue Sep 17, 2002 5:11 pm Post subject: really LARGE malloc

hullo,

I'm trying to allocate 2 GB of memory in a single malloc call in C. But it doesn't work, saying that I don't have enough memory.

I'm trying to find what is the real problem here (it is not memory, because I have 4 GB). Maybe it is a limit on the maximum a process can get. I tried to find something in the rlimit functions, but wasn't able to.

The kernel was compiled for 4 GB and it sees the memory. I understand that 512 MB is allocated to the kernel, as it doesn't appear in the memory summaries.

Thanks in advance.

phong · Posted: Tue Sep 17, 2002 6:05 pm Post subject:

If you're using x86 hardware, IIRC more than 2GB is probably not addressable as a single block (i.e. you're pushing architectual limitations). I'm not sure if that's the exact limit... Anyhow, I would see if you could break it into smaller pieces (i.e. have a few 1 gig mallocs) and see if that works... It might be impossible with current processors/kernels to address more than that in a single process.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer

pilla · Posted: Tue Sep 17, 2002 6:10 pm Post subject:

Yes, this was my first feelling, I couldn't remember the exacts limits of x86 hardware. I'm doing some research elsewhere. The next thing I'll try is a 64 GB memory kernel, maybe with a different memory model I'll be able to access all the memory (or at least 3 GB).

EDIT: seems that there is some problem with glibc/malloc. I can only allocate ~ 2 GB.

pilla · Posted: Tue Sep 17, 2002 8:57 pm Post subject:

OK, we have two limits here (according to my research)

First, Linux kernel separates 1 GB for the OS and 3 GB for user processes in order not to incur in performance loss (problems of x86 architecture).

Second, a process may have only 2 GB for it.

Please, shoot me if I'm saying something stupid/wrong.

rac · Posted: Tue Sep 17, 2002 9:08 pm Post subject:

I found this admittedly a bit old LK message from kernel hacker Rik van Riel, in which it claims that a process can allocate 3GB, as long as it uses mmap, if this helps.
_________________
For every higher wall, there is a taller ladder

pilla · Posted: Tue Sep 17, 2002 9:20 pm Post subject:

I've found this quite interesting document about Linux RAM limits.

Some interesting snippets:

* Linux 2.4.x does support greater then 4GB of physical memory (ie. RAM). It does this by supporting Intel's PAE (Physical Address Extension) features which are in all Pentium Pro and newer CPU's.

* PAE does not increase Linux's ability for *single* processes to see greater then 3GB of RAM (see below).

* With 2.4 kernels (with a large memory configuration) a single process can address up to the total amount of RAM in the machine minus 1GB (reserved for the kernel), to a maximum 3GB

As long as I could find out, newer mallocs use both mmap and brk (the later when it wants to allocate small chunks. mmap is more efficient for large chunks). There is also the problem of mapping shared libraries inside the process memory space, which reduces the memory that can be allocated even when it still has a 1 GB free

I'll try to keep this thread up-to-date with my findings (if I have some

)