Help tweak out our Opteron cluster!!!

[erikm]

Hi all,

I am about to design and install a 16 CPU Opteron Beowulf cluster at our research department. I run Gentoo x86/~x86 on a Pentium 4, a Pentium M, a dual Pentium 3 and another, smaller Athlon MP cluster, but I have not done any complete AMD 64 systems yet, just intermittent maintenance. Being short on time, I'd like to minimize the guesswork as much as possible, hence this thread.

I have the following general wishlist / questions for the OS, in order of priority:

1. Stability. I need a rock solid toolchain, preferrably as free from 32-bit emulation as possible. I have never had much trouble going ~x86 with the system on the other architectures in terms of stability, so what do you recommend, ~x86 or x86?

2. Optimization. Given a little more experience, I might well join the anti-ricer missionaries and preach the "i686 -O2" gospel to the dark side . However, every little tiny bit of performance increase counts here, and counts a lot; so go crazy (they'll have my ass for this ): What compiler flags do I choose?

3. Maintainability. It would be great to be able to schedule upgrades etc., and actually have them work, i.e. not having to constantly babysit the thing. Since we're talking Linux, and better still Gentoo Linux, there is a number of ways to achieve the same functionality with different software combinations. So, is there any packages I definitely should steer clear of?

4. Has anyone tried the new 2.6 OpenMosix patches on an Opteron?


Each slave node will consist of dual Opteron 248:s and 4 GB RAM on a Tyan MB, with dual Broadcom Gb NIC:s and a 36 GB 15 K rpm SCSI drive. The slaves need to run as optimized, light and free from process queues as possible. Additionally, network latency is paramount. My slave specific questions are thus as follows:

1. I'm thinking

[wazoo42]

Are you using fortran? If so you really want to steer clear of gcc b/c its -i8 flag does not work properly. Also, for all out performance you will probably get more improvement going to pgi or pathscale. I have tried both for electronic structure calcs (GAMESS) and they are speedy.

Are you installing 32 bit gentoo on these? I would guess not, but the x86 flags have me wondering. In terms of gcc I would say a good start would be to have -march=opteron -O2 -fomit-frame-pointer -funroll-loops -pipe.

[erikm]

Ah yes, I noticed I should have put amd64 as the architecture . So, do you suggest using icc or the pg c compiler? Where would I have used the -i8 flag? Thanks!

[wazoo42]

I have only used the demos, but pgi has much better support than pathscale. That being said, pathscale usually handily out performs pgi. The -i8 comes in when you want fortran to use 64bit integers.

[brankob]

OT, I know, but I'm curious- why are you not using dual-core Opterons , like 275?

That would mean only 4 motherboards instead of 8 and those are not exactly cheap, besides it could kill some of the net latency...

Unless of course you will be packing each CPU with as much ram as possible...

[visaris]

Depending on your needs, the dual-core parts may be a great way to go. I have a Tyan K8WE (s2895) with two Opteron 265s (dual-core @ 1.8GHz). This thing is fast.. I'll admit it took me just a little while to get it all working, but it was worth it.

As for basic advice, just be sure you get a MB that has ram banks connected to each CPU. Since each Opteron has it's own on-board memroy controller, make sure you get a motherboard that supports them all. I've seen some lame dual-proc boards that only use one mem controller.

[get sirius]

Just a point to make about your ventured USE flags - don't put those specific flags there. "sse" and "sse2" are both implied by the line in your make.conf, CHOST="x86_64-pc-linux-gnu", so they do not need to be mentioned at all. If you have processors built on the 90nm scale, then they have sse3 enabled. To make sure that gcc takes that into account, include "-msse3" in your CFLAGS line. And there is also no need to mention 3dnow, 3dnowext, or (and especially!)mmx.

I believe those use flags are included in the list of USE flags mainly for the use of developers.

[Albert_Alligator]

If you have a large amount of memory, using the -O3 flag is fine for HPC type computing, but you should test it out with both. A good benchmark is the HPC challenge benchmark. I will be posting a full tutorial of how to install and make it work soon.

GCC really isn't the compiler of high performance computing, although it will do in a pinch. If you're using g77 you should be fine, anything newer and you'll get into trouble with GCC.

Both Pathscale and PGI compilers are fast, but they are spendy, however, if you're willing to give up the Opteron boards, Intel has a very good compiler that if you don't sell it with a commercial system, you can have for free.

But as we all know, Intel CPUs don't hold a candle to the Opterons, so you'll have to make the choice yourself.

In a preliminary warm up test with GCC, I did the HPL test from the HPC challenge site and clocked a whopping 9.7 Gigaflops on a Tyan S4882 4-way board with 4 800 series Opterons.

Well, I'm tuning and I hope to break the 10 Gigaflop barrier, and who knows, maybe even get as high as 15 Gigaflops.

We'll have to wait and see.

Al
_________________
As Socrates once said "I drank what?"

[erikm]

[Desti²]

Some POVRAY Benches GCC vs. Icc: http://pov4grasp.free.fr/articles/fastpov1/
_________________
Linux Users Everywhere @ climateprediction.net

[dweigert]

Folks,
Just remember that the Intel compiler suite has been intentionally crippled by checking for Genuine Intel, before checking the processor capability flags.
Any other manufacturer's CPU will be routed to the *Slowest* code paths, including ones that truly are brain damaged.

Dan
_________________
"Always remember to mount a scratch monkey..."

[erikm]

That doesn't change the fact that gcc obviously is the better choice for an Opteron, though.

[nephros]

[piwacet]

http://www.swallowtail.org/naughty-intel.html

[nephros]