Gentoo cluster install questions.

Message

pklizest · Post by **pklizest** » Wed Dec 22, 2010 7:33 am

Hi, I have been wanting to install setup a Gentoo cluster for sometime, but I have a collection of about 30 odd computers with various broken bits that have been donated to me, and this has led me to reinstall Gentoo multiple times on these box. However this is painfully slow due to the Internet connection at my lab being junk. I was hoping there is a relatively easy way to slap a base install on the various computers quickly without being forced to install manually each time. I would then go back in and modify stuff as needed. I have both 32bit and 64bit 86x archs computers, and I also have some old sun servers.

I was going to add the latests portage and stage3 to the minimal Gentoo iso, and put a stripped down version of glis on the cd as well. But before I make a few different isos learn how to bash script and refactor iso properly, I wanted to ask if anyone had a better way to do this.

tl;dr, How do i unattended install gentoo with no internets?

glis=Gentoo Linux install script; http://glis.sourceforge.net/

pappy_mcfae · Post by **pappy_mcfae** » Wed Dec 22, 2010 8:34 am

Sure. Set up one machine with a generic Gentoo install.

Once the install is done, do a stage-4 to it, and use the stage-4 to populate the other computers. That way, you only have to do the hard work once. With the stage-4, you copy your finished system to all the machines. Once that's done, just customize the kernels and make.conf, and go from there.

What is a stage-4? It is a form of a backup that uses tar to create a tarball. The tarball created is a working Gentoo installation. A working Gentoo installation is defined as a fully functional base Gentoo operating system one will have when done following the Gentoo Handbook. By creating the backup and distributing it to the other machines, all you have to do is to untar, chroot, setup the boot loader, and you're done.

It saves a lot of time...believe me.

Cheers,
Pappy

delenda · Post by **delenda** » Tue Dec 28, 2010 6:56 pm

A few questions...How is your cluster physically interconnected? By switches, routers (locally but fast) ? Or direct to the internet ?
Perhaps some computers can be booted through LAN? Diskless install using PXE boot

Do you require 64bit software, or could everything be 32bit?

[Edit]
Just found this: http://www.gentoo-wiki.info/HOWTO_Create_A_Build_Host

delenda · Post by **delenda** » Sun Jan 02, 2011 7:31 pm

[Edit - 2: must be talking to meself]

I'm trying to build my own cluster..I don't have like "about 30 odd computers", more like 03...but generally, it's the same principle.

Now here's an idea, create a general live-cd (or dvd usb-stick/hdd/...) on your main (quickest) computer...
(see links)...Either setup an local DNS server identifying each node (dhcp/MAC adress), or adapt the basic image, e.g. change every host name, et cetera, before imaging it to the cd/dvd/usb-stick....Then boot the node from its personalized LiveCD/LiveUSB/LiveDVD....

Each node would adress the server and log on..The cluster grows....

Some thoughts: if I would were to use my most powerful node only, leaving the remaining nodes unused, I'd spare the electric bill.

One out of my 03 nodes, has now depreciated hardware, IDE but still a fully functional hard drive

So, if you do have "odd hardware" - hmmm

30odd computers, so probably - , why bother to install it "fully optimized"? I mean CFLAGS like -march=pentium4, would only complicate everthing.

One could alter the livecd to boot and expand into ram, thus utilizing only cpu-ram-eth, omitting any other "odd" hardware. Heck linux doesn't use much ram, as win#!#. A stripped version - bootable toolchain, no X, gnome,gtk,kde ..- uses far less..

Included bonus, a minimal "livecd-boot" would be easier to maintain. A new toolchain could destroy a outdated compilation farm. Updating a few livecd-stages, burning those to discs/sticks would save a lot of time in stead of duplicating this to each nodes hard disc....

If you really want to recuperate some of the 30-odd computer hardware - eg huge disc space - , one could easily focus those boot-discs. (eg disc - nfs server posibilities - or set it up locally...)

Another idea: since most of those nodes would not be used graphically, one could easily tap into it's graphic abilities, I mean ram...I'd use it as tmp or as as a ramdisc containing the toolchain....swapping would probably not be really used.

Links:
http://en.gentoo-wiki.com/wiki/Build_Yo ... or_LiveDVD
http://forums.gentoo.org/viewtopic-t-296892.html
http://en.gentoo-wiki.com/wiki/Using_Gr ... ry_as_Swap

PS: I only used a stage2/stage3 installation
PPS: I'm currently unsuccessful building an 1 personalized LiveCD, but still trying...

Raptor85 · Post by **Raptor85** » Wed Jan 05, 2011 1:17 am

pappy's way is the way to go, after that you're going to want to set up a binhost on one of your machines so you only need compile once. If they're identical machines you're golden, if not, you'll want to carefully check all drivers needed for each machine, and the "common" architecture (probably 686), you'll lose performance somewhat on machines that support more, but it will save you a lot of time and leave you with only one common image to maintain.

I have distcc set up across all of mine, one of these days i need to test it by re-compiling openoffice across all the nodes. I wonder if there's a record for fastest compile of that yet

pgu · Post by **pgu** » Fri Jan 07, 2011 2:36 pm

Is there a way to run all the nodes in a cluster diskless and share all the binaries (assuming all x86_64 compatible nodes)?
I used to do this in the good old days using SunOS. /var, /tmp, /etc was unique as /bin/ /lib etc was share among all the nodes in the cluster.

That way adding a node in the cluster would be simply to duplicate the unique part on the server, enable PXE in the new node and boot.

pgu · Post by **pgu** » Fri Jan 07, 2011 2:46 pm

delenda wrote: PPS: I'm currently unsuccessful building an 1 personalized LiveCD, but still trying...

I built an install CD on a USB stick once, just followed the instructions on the wiki I beleive. It had sshd enabled by default with my key in /root/.ssh/authorized_keys. I could then ssh straight into the machine and do the install.

Some clever expect script or similar could probably automate the installation.

delenda · Post by **delenda** » Sat Jan 08, 2011 11:27 am

Building a stage-4 and extracting it to the local hard disc, or creating a bootable disc/stick is basically the same...

Only the destination differs.

wim-x · Post by **wim-x** » Sat Jan 08, 2011 12:09 pm

pgu wrote:Is there a way to run all the nodes in a cluster diskless and share all the binaries (assuming all x86_64 compatible nodes)?

Yes, this is possible, I do this myself. For this purpose I made a chroot to which each thin-cluster-client can boot through nfs.

pgu · Post by **pgu** » Sun Jan 16, 2011 6:18 pm

wim-x wrote:Yes, this is possible, I do this myself. For this purpose I made a chroot to which each thin-cluster-client can boot through nfs.

What parts of the file-system do you share, and which ones do you not share (at least /tmp, /var/, /etc)?

wim-x · Post by **wim-x** » Tue Jan 18, 2011 6:53 pm

pgu wrote:What parts of the file-system do you share, and which ones do you not share (at least /tmp, /var/, /etc)?

Everything is shared, but some parts are mounted as tmpfs. I think /tmp /var/tmp /var/run /var/log /var/spool are the most important.

I was quite lazy actually; I'm using LTSP for the process to create the thin client environment. In this setup, I'm using a customized Quickstart profile file to not install a complete LTSP client, but a minimal thin client environment. In the thin client boot process, the ltsp-client-setup does all the tmpfs mounting.

While this was quite easy, it's not the most elegant way

pgu · Post by **pgu** » Tue Jan 18, 2011 7:51 pm

I would assume /etc being unique due to /etc/fstab, /etc/hostname etc?

I would like to set up a front-node which serves a cluster of nodes to be used for simulations. As I overclock the nodes they tend to be disposed pretty rapidly, hence I would like to make the installation as easy as possible. Enable PXE or insert an USB stick and boot, then reformat the local hard drives which are used for scratch storage during simulation. Linux and software resides on the front-node.

pgu · Post by **pgu** » Tue Jan 18, 2011 7:53 pm

At home I have a couple diskless mini-itx machines. They boot from the server, and nfs mount the file system, but nothing is shared among the clients in this setup. Each machine have their own file system tree on the server.

But I would like to share to make it easier to add nodes to the cluster. New software would be available to all nodes.

wim-x · Post by **wim-x** » Thu Jan 20, 2011 5:32 pm

pgu wrote:I would assume /etc being unique due to /etc/fstab, /etc/hostname etc?

Could be I guess, never tried that. As I said, I'm using a custom LTSP thin client chroot.

pgu wrote:But I would like to share to make it easier to add nodes to the cluster. New software would be available to all nodes.

This is exactly how I manage my thin clients. When adding a new node, I just have to configure the dhcp server (dnsmasq) and the pxelinux config with the new mac and ip.