Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NUMA for unganged memory on a Phenom II?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
piwacet
Guru
Guru


Joined: 30 Dec 2004
Posts: 486

PostPosted: Wed Dec 09, 2009 2:54 am    Post subject: NUMA for unganged memory on a Phenom II? Reply with quote

I realized my computer can run the memory in two modes: 'ganged' - where all 4 cores of my single phenom II utilize both memory controllers together for 128-bit memory access, i.e. as if the phenom II had a single 128-bit wide memory controller that all 4 cores would try to use (but not be able to use simultaneously); and 'unganged' mode, where each of the 2 memory controllers acts independently in 64-bit mode, one serving 2 cpus, and the other memory controller serving the other 2 cpus. 'Unganged' mode permits simultaneous memory access by both controllers, although at a lower memory bandwith. Apparently AMD recommends the 'unganged' mode, and some benchmarks support using unganged for multithreading.

To me, unganed sounds like numa - 2 cpu's work through 1 memory controller that is assigned it's own stick of RAM; and the other 2 cpus work through the other memory controller which accesses the other stick of RAM. ( I currently have 2 sticks.)

If I'm understanding this correctly, making the kernel NUMA aware would be a good idea. Does anyone know if this is recommended or have any experience with this? I may already have the answer - I did enable NUMA support, and dmesg essentially says 'no NUMA configuration found' and proceeds to fake it with only one node - which I assume is useless. But I was hoping someone would have some other insight...

Thanks!
Back to top
View user's profile Send private message
gentoo_ram
Guru
Guru


Joined: 25 Oct 2007
Posts: 398
Location: San Diego, California USA

PostPosted: Wed Dec 09, 2009 6:29 am    Post subject: Reply with quote

No, that is not NUMA. NUMA is used on much larger boxes. It's very uncommon on anything you would put in a home. I don't know of it being used on any box worth less than, say, $250K. I don't know what that AMD mode thingie is, but it definitely isn't NUMA. Maybe your BIOS settings say something about that mode?

The one box I knew about that had NUMA was a 16 or 32-processor Alpha processor server. Any CPU could see any of the memory. But they were 4-processor cages connected via a slower bus. Thus any memory outside of your little cage was much slower to the CPU. That's NUMA. I bet there are other such boxes that use NUMA as well.
Back to top
View user's profile Send private message
piwacet
Guru
Guru


Joined: 30 Dec 2004
Posts: 486

PostPosted: Wed Dec 09, 2009 7:16 am    Post subject: Reply with quote

Thanks. I've been thinking about this, and my assumption that each controller services it's own respective CPUs may not be right; I really don't know. In other words, if the computer is running in unganged mode, it may be that each core still has equal access to all DIMMS - they would just need to access the appropriate memory controller that's running that DIMM. If this is true, there is really no role for NUMA - this would not be NUMA.

In other words, suppose I have two DIMMs, DIMM1 and DIMM2. Memory controller 1 (MC1) works with DIMM1, and Memory controller 2 (MC2) works with DIMM2.

In unganged mode, if a thread is running on CPU1, and has it's data stored in DIMM1, it would be talking to MC1 to get data from memory. If this thread was rescheduled to any of the other 3 CPU's, they all may still be able to talk to MC1 directly to get the data off DIMM1, as opposed to having to move the data from DIMM1 to DIMM2 (which could be necessary if cores were assigned to individual memory controllers, depending on which core the thread ended up on). If this is how it works, there are no NUMA issues.

I don't know, but this makes sense to me.

Thanks for the help!
Back to top
View user's profile Send private message
aguettinger
Tux's lil' helper
Tux's lil' helper


Joined: 26 Sep 2003
Posts: 84

PostPosted: Wed Dec 09, 2009 1:44 pm    Post subject: Reply with quote

gentoo_ram wrote:
No, that is not NUMA. NUMA is used on much larger boxes. It's very uncommon on anything you would put in a home. I don't know of it being used on any box worth less than, say, $250K. I don't know what that AMD mode thingie is, but it definitely isn't NUMA. Maybe your BIOS settings say something about that mode?

The one box I knew about that had NUMA was a 16 or 32-processor Alpha processor server. Any CPU could see any of the memory. But they were 4-processor cages connected via a slower bus. Thus any memory outside of your little cage was much slower to the CPU. That's NUMA. I bet there are other such boxes that use NUMA as well.


That is not correct I think. Actually any AMD Opteron supports NUMA.

http://www.cs.utk.edu/~vose/linux/NUMA.html
_________________
regards
- andy
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum