Make your divX with a cluster of machines WITH EASE

[fxlamare]

Hi everyone !

I'm setting up a system based on ltsp (www.ltsp.org) and dvdrip (http://www.exit1.org/dvdrip ) to allow the making of a divX to be done with several machines.

i already did some prototypes servers, and it works rather well, but i have to clarify the server' setup procedure

to precise things, the idea is to boot machines from the network (U don't have to install linux on them). Assume those machines are the NODES of our cluster. think of rather fast machines, at least PIII800 MHz. For best results, nodes shouldn't have too different CPUs : you won't have any improvment if you make a cluster with a Pentium II 300 based node and an athlon XP 2200+ based one !!!

the big advantage of this trick is that when everything is setup (ok that's not easy to achieve and quite long at the moment !) you simply need an etherboot floppy (or a PXE capable NIC) to boot nodes. You can add as many nodes as you want. OK the network may become quite busy, i'll make tests to know the bandwidth used by a node...

the BIG problem is that the system provided with the ltsp's project has just the necessary to afford nodes to connect to an X server. It's nearly impossible to add packages to their system (as for version 3 of the project) and I must add dvdrip and ssh for my cluster to work...(see dvdrip's cluster-mode requirements for details)

Anyway, I achieved to prepare a prototype, simply by replacing the ltsp's 'root system' by a stage3-i686-gentoo-based system, into which I include LTSP's boot scripts and binaries.

then I use portage as usual in this system (chrooted environment) to compile all the needed packages.

the most time-consuming step is the ltsp boot scripts analysis, and the adaptation of the gentoo's tree to comply with an read only file system, exported via NFS....

but it works...

protocols to setup
dhcp | tftp | NIS | NFS | SSH

NFS exported directory are the root FS, /home and a /dvd directory containing the ripped DVD (.vob files)

when all is setup up :
I boot nodes with etherboot images (www.rom-o-matic.net) in runlevel 3
I start dvdrip on the master server
I set the project properties and add it to the cluster daemon
I add my nodes
I launch the process...

i must make further tests before I publish results...
if you have any comment on this.. or idea??

and sorry for my average english...

F.X. LAMARE
LILLE, France : 133 miles north of PARIS
and... don't ask me how i got this idea...

[Evangelion]

Shouldn't this go to "Documentation, tips & Tricks" ?

About the network-booting... How exactly do you do it? To my knowledge, you have three choices with LTSP:

-boot-floppy
-NIC with boot-ROM
-HD

I'm interested in thin-clients, but I need a good way to boot the machines. HD is not OK (drives up the cost and I want a thin-client with no moving parts), as is floppy (same reasons as with HD's), so I have the NIC-option left. But don't you need NIC's with custom boot-ROM in order to work with LTSP? What would be the best way to make the clients boot from the server?
_________________
My tech-blog | My other blog

[Matje]

I did (sorta) the same thing to create my thin-game-clients for lanparty's, it uses a 100 Mb switch and it can cope with all servers (4 - 5 or more ) running at the same time quite happily. I started experimenting with ltsp but it evolved to what you got, I only use the initrd (which I modified) and the patch for the kernel of the ltsp-package. Furthermore I had to update a lot of init-scripts to cope with some problems...
_________________
Life is like a box of chocolates... Before you know it, it's empty...

[Visnu]

Hmmm, if you could do this to make DivX, would you be able to setup multiple machines to compile the same packages? Imagine how quickly 3-4 machines could emerge kde or gnome.

[noff]

[fxlamare]

[Evangelion]