SPARSEMEM vs. FLATMEM (Solved)

Message

causality · Post by **causality** » Thu Apr 07, 2011 4:51 pm

In my kernel config under "Processor Type and Features" I now have a choice to make for "Memory Model".

For the longest time, "Flat Memory" was the only available option. Recently, I can also choose "Sparse Memory". These are the only two options available to me. Flat memory is the default and it's what I have been using for a long time now, but I am curious about sparse memory and whether I should try it.

I am having a hard time finding a real, solid definition of what sparse memory is. Near as I can tell, it's like sparse files in that it may be non-contiguous and may have holes in its allocation. But that leaves some things unanswered. It doesn't tell me if SPARSEMEM is a drop-in replacement for FLATMEM that won't break anything. It doesn't tell me what performance differences there may be, if any. If it helps, this is a standard desktop PC with an AMD64 X2 processor; I have nothing exotic in terms of hardware.

Any attempt to Google this topic results in tons of posts from various Linux Kernel mailing lists where people are talking about how to implement it in C without discussing its merits. Any other search results just regurgitate the "help" text from the kernel config, something a lot of sites like to repost endlessly without adding new information to it. That makes it difficult for those who are seaching in the first place because they need more information than provided by the help text they already have.

At any rate, it's a surprisingly difficult and not-straightforward topic to try to research.

The kernel config help text does suggest selecting FLATMEM if you are in doubt, but it doesn't explain why.

Anyway, I was hoping someone here who is knowledgable about these things might be willing to help me understand the difference.

aCOSwt · Post by **aCOSwt** » Thu Apr 07, 2011 5:05 pm

The only measure I know about this topic was at the time sparsemem was challenging discontig :

Code: Select all

Measurements in cycle counts. 1000 memory allocations are performed and then the
average cycle count are calculated.

Order	FlatMem	Discontig	SparseMem
0	  639	  665		  641
1	  567	  647		  593
2	  679	  774		  692
3	  763	  967		  781
4	  961	 1501		  962
5	 1356	 2344		 1392
6	 2224	 3982		 2336
7	 4869	 7225		 5074
8	12500	14048		12732
9	27926	28223		28165
10	58578	58714		58682

This was showing some significant interest to sparsemem over discontig when flatmem producing a non significant advantage over sparsemem.

For what it's worth... dating from about 5 years ago...

Post by **NeddySeagoon** » Thu Apr 07, 2011 5:53 pm

causality,

Sparse memory is only of use on NUMA systems or systems supporting hot pluggable memory.

A NUMA system supports several CPUs (not CPU cores) each with their own local memory and a way for each CPU to addess the global memory pool. Accessing the non-local memory is usually slower.

When memory is hot plugged, the kernel determines where in the physical address space it goes.

Both features make a big hole in your wallet - you would know if you had either of them.

causality · Post by **causality** » Thu Apr 07, 2011 6:31 pm

You just provided what a great deal of searching did not. Thank you for giving me a truly helpful answer.

I will stick with FLATMEM since it sounds like SPARSEMEM wouldn't work on my regular PC.

aCOSwt · Post by **aCOSwt** » Thu Apr 07, 2011 6:58 pm

NeddySeagoon wrote: Sparse memory is only of use on NUMA systems or systems supporting hot pluggable memory.

Not willing to argue in any manner, just wishing to understand something I am getting confused with since your post.

Are we speaking about CONFIG_SPARSEMEM_MANUAL ?

In which case this is what I get for my Gentoo-Sources 2.6.34-r12 running on a standard (Non NUMA) x86_64 SMP arch :

Code: Select all

Symbol: SPARSEMEM_MANUAL [=y]
Depends on: <choice> && ARCH_SPARSEMEM_ENABLE [=y]

Symbol: ARCH_SPARSEMEM_ENABLE [=y]
Selects: SPARSEMEM_STATIC [=n] && SPARSEMEM_VMEMMAP_ENABLE [=y]

Symbol: SPARSEMEM_VMEMMAP_ENABLE [=y] 
Selected by: ARCH_SPARSEMEM_ENABLE [=y] && (X86_64 [=y] || NUMA [=n] || EXPERIMENTAL [=y] && X86_32 [=n] || X86_32_NON_STANDARD [=n]) && X86_64[=y]

Moreover... it is the only available option because :

Code: Select all

Symbol: FLATMEM_MANUAL [=n] 
Depends on: <choice> && (!ARCH_DISCONTIGMEM_ENABLE [=n] && !ARCH_SPARSEMEM_ENABLE [=y] || ARCH_FLATMEM_ENABLE [=n])

CONFIG_SPARSEMEM_MANUAL appears not linked with NUMA or other hot-pluggable memory option, it even appears to be the only option available as soon as ARCH_SPARSEMEM_ENABLE is set. (This behind my back...)

Post by **NeddySeagoon** » Thu Apr 07, 2011 9:09 pm

aCOSwt,

My 2.6.38 is aet up as

Code: Select all

CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y

Not all those options are user visible. Memuconfig shows I have sparse memory select as the only choice too.

Having Spare Memory set (or forced on you) does not mean your RAM cannot be contiguious, only that it need not be.
Some time ago, I had both options available.

1clue · Post by **1clue** » Thu Apr 07, 2011 9:12 pm

NeddySeagoon wrote:causality,

Sparse memory is only of use on NUMA systems or systems supporting hot pluggable memory.

A NUMA system supports several CPUs (not CPU cores) each with their own local memory and a way for each CPU to addess the global memory pool. Accessing the non-local memory is usually slower.

When memory is hot plugged, the kernel determines where in the physical address space it goes.

Both features make a big hole in your wallet - you would know if you had either of them.

Seriously.

This needs to go into the kernel documentation.

Post by **NeddySeagoon** » Thu Apr 07, 2011 9:18 pm

1clue,

I think that's where I found it - but it was a few years ago now.

Edit: See /usr/src/linux/Documentation/memory-hotplug.txt

1clue · Post by **1clue** » Thu Apr 07, 2011 9:21 pm

I KNOW I have clicked the help on that feature nearly every time through, but never got the message as clearly as when you said it.

Maybe they just need to add the part about the hole in your wallet.