Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
poor-man's distributed parallel computing framework ?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sat Jan 05, 2013 8:45 am    Post subject: poor-man's distributed parallel computing framework ? Reply with quote

Hi,

I'm looking for a simple yet hackable framework to build a tiny cluster. Has anyone seen something like this?
Features I'm looking for:

  • hot-add of new nodes
  • job constraints - need to run certain processes on only certain nodes, e.g. they have special hardware connected
  • ssh backend integration - need to hot-add nodes really quickly and my colleagues don't like to learn new technologies
  • cross-platform (at least the client) if possible, with easy deployment
  • python interface would be nice
  • job dependencies - at least before/after-like keywords

Background: I'm trying to make a better data processing system for our small tokamak reactor and as we implement more sophisticated analysis SW, the CPU and memory cost rises. I need a core cluster with a few nodes for the basic analysis, but then from time to time a student comes with some special hardware that can be controlled through e.g. his notebook and there's no time to install it on the main servers, so giving him an ssh account+key for uploading the data is usually the simplest way.

I've already done some research, but IPython seems too complex and doesn't have ssh integration (only ssh tunnels) and GNU parallel doesn't have dependencies. I feel like I'm going to have to write most of the integration on my own, but a few helpful libraries could speed it up.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1565
Location: U.S.A.

PostPosted: Sat Jan 05, 2013 10:16 am    Post subject: Reply with quote

Beowulf?
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sat Jan 05, 2013 10:38 am    Post subject: Reply with quote

Thanks BK, I remember seeing the name but AFAIK it's more of an abstract term describing a poor-man's cluster, almost like a specification.
The specification (if there even is something like that) appears to be very vague and I cannot find any simple deployment docs.
Could you recommend any relevant and up-to-date docs on how to make a Beowulf cluster?
So far I've found OSCAR and openMosix(looks dead) for management, could you recommend any other?

The trouble is that it behaves like one big supercomputer, I need a bit more granularity due to specific hardware dependencies, but I could use at least some of the technologies. Also, I may need to hot-add a computer and need to deploy it quickly, without installing any special SW if possible.

EDIT: Thinking about it, our present setup could be described as a Beowulf cluster, but sorta patched up, I'm looking for something more automated for management.
EDIT2: any experience with TORQUE?
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1565
Location: U.S.A.

PostPosted: Sat Jan 05, 2013 11:18 am    Post subject: Reply with quote

No, sorry.

Random link:
http://idea.uab.es/mcreel/PelicanHPC/
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sat Jan 05, 2013 11:24 am    Post subject: Reply with quote

Thx BK, looks interesting, but a live system isn't really what I need.

Torque caught my eye, anyone have some interesting experience with it? Or any superior SW that you could recommend?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16112
Location: Colorado

PostPosted: Sat Jan 05, 2013 5:18 pm    Post subject: Reply with quote

Not sure about capabilities, but I've read about Rocks Cluster Distribution.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sat Jan 05, 2013 10:27 pm    Post subject: Reply with quote

Thx for the tip pjp, it's mainly targeted at HPC, more like PVE. I need something more granular.
Looking into Torque now.
Back to top
View user's profile Send private message
pigeon768
l33t
l33t


Joined: 02 Jan 2006
Posts: 669

PostPosted: Sun Jan 06, 2013 11:46 am    Post subject: Reply with quote

In addition to torque, you might check out slurm.
_________________
My political bias.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Jan 06, 2013 12:20 pm    Post subject: Reply with quote

thank you pigeon768, SLURM looks also interesting as an alternative.
However, it requires each node to have slurmd running as root (pbs_mom in torque does too, unfortunately), which is a bit too demanding in our case.
I need to be able to give a user a few packages and scripts that he can quickly deploy on his computer/server, where he may not have root.
Torque got close with their neat package distribution system, but the root privileges requirement is a show stopper... :(
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1565
Location: U.S.A.

PostPosted: Sun Jan 06, 2013 12:31 pm    Post subject: Reply with quote

This is starting to sound a little like "grid computing" (i.e. "distributed computing" as opposed to "clusters"). I'm not sure what you're trying to do, but have you looked at BOINC?
_________________
pjp wrote:
I didn't misquote you, I just misunderstood you.


Last edited by Bones McCracker on Sun Jan 06, 2013 1:02 pm; edited 2 times in total
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Jan 06, 2013 12:34 pm    Post subject: Reply with quote

nice find BK, will have a look at it after lunch.
Looks promissing, should have simple deployment, now the question is how granular their job manager is...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum