NUMA for unganged memory on a Phenom II?

[piwacet]

I realized my computer can run the memory in two modes: 'ganged' - where all 4 cores of my single phenom II utilize both memory controllers together for 128-bit memory access, i.e. as if the phenom II had a single 128-bit wide memory controller that all 4 cores would try to use (but not be able to use simultaneously); and 'unganged' mode, where each of the 2 memory controllers acts independently in 64-bit mode, one serving 2 cpus, and the other memory controller serving the other 2 cpus. 'Unganged' mode permits simultaneous memory access by both controllers, although at a lower memory bandwith. Apparently AMD recommends the 'unganged' mode, and some benchmarks support using unganged for multithreading.

To me, unganed sounds like numa - 2 cpu's work through 1 memory controller that is assigned it's own stick of RAM; and the other 2 cpus work through the other memory controller which accesses the other stick of RAM. ( I currently have 2 sticks.)

If I'm understanding this correctly, making the kernel NUMA aware would be a good idea. Does anyone know if this is recommended or have any experience with this? I may already have the answer - I did enable NUMA support, and dmesg essentially says 'no NUMA configuration found' and proceeds to fake it with only one node - which I assume is useless. But I was hoping someone would have some other insight...

Thanks!

[gentoo_ram]

No, that is not NUMA. NUMA is used on much larger boxes. It's very uncommon on anything you would put in a home. I don't know of it being used on any box worth less than, say, $250K. I don't know what that AMD mode thingie is, but it definitely isn't NUMA. Maybe your BIOS settings say something about that mode?

The one box I knew about that had NUMA was a 16 or 32-processor Alpha processor server. Any CPU could see any of the memory. But they were 4-processor cages connected via a slower bus. Thus any memory outside of your little cage was much slower to the CPU. That's NUMA. I bet there are other such boxes that use NUMA as well.

[piwacet]

Thanks. I've been thinking about this, and my assumption that each controller services it's own respective CPUs may not be right; I really don't know. In other words, if the computer is running in unganged mode, it may be that each core still has equal access to all DIMMS - they would just need to access the appropriate memory controller that's running that DIMM. If this is true, there is really no role for NUMA - this would not be NUMA.

In other words, suppose I have two DIMMs, DIMM1 and DIMM2. Memory controller 1 (MC1) works with DIMM1, and Memory controller 2 (MC2) works with DIMM2.

In unganged mode, if a thread is running on CPU1, and has it's data stored in DIMM1, it would be talking to MC1 to get data from memory. If this thread was rescheduled to any of the other 3 CPU's, they all may still be able to talk to MC1 directly to get the data off DIMM1, as opposed to having to move the data from DIMM1 to DIMM2 (which could be necessary if cores were assigned to individual memory controllers, depending on which core the thread ended up on). If this is how it works, there are no NUMA issues.

I don't know, but this makes sense to me.

Thanks for the help!

[aguettinger]

No, that is not NUMA. NUMA is used on much larger boxes. It's very uncommon on anything you would put in a home. I don't know of it being used on any box worth less than, say, $250K. I don't know what that AMD mode thingie is, but it definitely isn't NUMA. Maybe your BIOS settings say something about that mode?

The one box I knew about that had NUMA was a 16 or 32-processor Alpha processor server. Any CPU could see any of the memory. But they were 4-processor cages connected via a slower bus. Thus any memory outside of your little cage was much slower to the CPU. That's NUMA. I bet there are other such boxes that use NUMA as well.

That is not correct I think. Actually any AMD Opteron supports NUMA.

http://www.cs.utk.edu/~vose/linux/NUMA.html