poor-man's distributed parallel computing framework ?

smartass

Hi,

I'm looking for a simple yet hackable framework to build a tiny cluster. Has anyone seen something like this?
Features I'm looking for:

hot-add of new nodes
job constraints - need to run certain processes on only certain nodes, e.g. they have special hardware connected
ssh backend integration - need to hot-add nodes really quickly and my colleagues don't like to learn new technologies
cross-platform (at least the client) if possible, with easy deployment
python interface would be nice
job dependencies - at least before/after-like keywords

Background: I'm trying to make a better data processing system for our small tokamak reactor and as we implement more sophisticated analysis SW, the CPU and memory cost rises. I need a core cluster with a few nodes for the basic analysis, but then from time to time a student comes with some special hardware that can be controlled through e.g. his notebook and there's no time to install it on the main servers, so giving him an ssh account+key for uploading the data is usually the simplest way.

I've already done some research, but IPython seems too complex and doesn't have ssh integration (only ssh tunnels) and GNU parallel doesn't have dependencies. I feel like I'm going to have to write most of the integration on my own, but a few helpful libraries could speed it up.

BoneKracker · Veteran Joined: 14 Mar 2006 Posts: 1487 Location: U.S.A.

Beowulf?
_________________
Oldthinkers unbellyfeel INGSOC.
-- Headline of a document on Winston Smith's terminal in his cubicle at the Ministry of Truth, seen briefly in the background in one scene of the movie rendition of Nineteen Eighty-Four.

smartass · Posted: Sat Jan 05, 2013 10:38 am Post subject:

Thanks BK, I remember seeing the name but AFAIK it's more of an abstract term describing a poor-man's cluster, almost like a specification.
The specification (if there even is something like that) appears to be very vague and I cannot find any simple deployment docs.
Could you recommend any relevant and up-to-date docs on how to make a Beowulf cluster?
So far I've found OSCAR and openMosix(looks dead) for management, could you recommend any other?

The trouble is that it behaves like one big supercomputer, I need a bit more granularity due to specific hardware dependencies, but I could use at least some of the technologies. Also, I may need to hot-add a computer and need to deploy it quickly, without installing any special SW if possible.

EDIT: Thinking about it, our present setup could be described as a Beowulf cluster, but sorta patched up, I'm looking for something more automated for management.
EDIT2: any experience with TORQUE?

BoneKracker · Veteran Joined: 14 Mar 2006 Posts: 1487 Location: U.S.A.

No, sorry.

Random link:
http://idea.uab.es/mcreel/PelicanHPC/
_________________
Oldthinkers unbellyfeel INGSOC.
-- Headline of a document on Winston Smith's terminal in his cubicle at the Ministry of Truth, seen briefly in the background in one scene of the movie rendition of Nineteen Eighty-Four.

smartass · Posted: Sat Jan 05, 2013 11:24 am Post subject:

Thx BK, looks interesting, but a live system isn't really what I need.

Torque caught my eye, anyone have some interesting experience with it? Or any superior SW that you could recommend?

pjp · Posted: Sat Jan 05, 2013 5:18 pm Post subject:

Not sure about capabilities, but I've read about Rocks Cluster Distribution.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008

smartass · Posted: Sat Jan 05, 2013 10:27 pm Post subject:

Thx for the tip pjp, it's mainly targeted at HPC, more like PVE. I need something more granular.
Looking into Torque now.

pigeon768 · l33t Joined: 02 Jan 2006 Posts: 667

In addition to torque, you might check out slurm.
_________________
My political bias.

smartass · Posted: Sun Jan 06, 2013 12:20 pm Post subject:

thank you pigeon768, SLURM looks also interesting as an alternative.
However, it requires each node to have slurmd running as root (pbs_mom in torque does too, unfortunately), which is a bit too demanding in our case.
I need to be able to give a user a few packages and scripts that he can quickly deploy on his computer/server, where he may not have root.
Torque got close with their neat package distribution system, but the root privileges requirement is a show stopper...

BoneKracker · Veteran Joined: 14 Mar 2006 Posts: 1487 Location: U.S.A.

This is starting to sound a little like "grid computing" (i.e. "distributed computing" as opposed to "clusters"). I'm not sure what you're trying to do, but have you looked at BOINC?
_________________
Oldthinkers unbellyfeel INGSOC.
-- Headline of a document on Winston Smith's terminal in his cubicle at the Ministry of Truth, seen briefly in the background in one scene of the movie rendition of Nineteen Eighty-Four.

smartass · Posted: Sun Jan 06, 2013 12:34 pm Post subject:

nice find BK, will have a look at it after lunch.
Looks promissing, should have simple deployment, now the question is how granular their job manager is...