single system boot image

srd · n00b Joined: 13 Apr 2010 Posts: 38

Anyone familiar with how one might setup a few diskless nodes using a single system image? Or any known packages similar to openssi that work with gentoo? I'd like to have a single file system from which all diskless nodes boot as opposed to the gentoo diskless howto which is a complete copy of a file system per node, terribly inefficient.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

Make the server export the diskless root read-only so that no clients can accidentally mutate it. Configure all clients via DHCP. If you have any per-machine configuration, consider using an initramfs that can overmount a host-specific tmpfs with the relevant files before it begins booting the shared /sbin/init.

srd · n00b Joined: 13 Apr 2010 Posts: 38

Diskless root read-only? I have /usr, /var, etc... on separate partitions as well, so I assume you mean those too since the diskless nodes need them as well.

How do you handle p/ machine configuration files like:
- /etc/hosts
- diskless nodes that store pid files in the same location /var/run/my-app/my-app.pid
- each node having it's own set of init scripts from rc-update

Can you elaborate on how to do the per-machine configuration a bit more using initramfs. I've not done this before.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

Yes, for safety, you should make every shared export read-only. You could probably find a way to make multiple machines coexist with all of them writing to the exports, but it is unnecessary complexity.

You do not need /etc/hosts to be per-machine. Instead, let all systems have the same file and have it name all the machines.
You have two options for directories under /var. If you do not need to persist their contents, you could use a tmpfs for the area, and have the initramfs create the directory layout. If you want to persist the contents, or if you have insufficient RAM to create such a setup, you could give each machine a private /var on the server. Regardless, you can use a tmpfs for /var/run, /var/lib/init.d, and any other areas where the system writes files that are only meaningful until the system reboots.
For init scripts, /etc/conf.d settings, and any other situations where you need a truly per-machine configuration, you will need to have those prepared during boot. If an entire directory is per-machine, you can mount a tmpfs there and place contents accordingly. If only a few files need to be specialized, replace that file on the server with a symlink that will point to a location that is overmounted with per-machine state.

Once you have identified the directories which need to be customized, you need to add a block to your initramfs /init that will place a writable filesystem on those directories. You can use a tmpfs if writing to the disk is not allowed, as in this scenario. The initramfs then must populate the resulting area. The easiest way to do this would be with a pre-made tarball on the server. To take a hint from pxelinux, consider having the initramfs search for a tarball named after the MAC address of the primary interface:ip l show dev $(ip r | gawk '/default via / { print $5; }') | gawk '/link\/ether / { print $2; }'. If such a tarball is not found, begin stripping off octets or otherwise simplifying the name until you reach a default name that will hold the fallback configuration used by machines which have no specific instructions.

srd · n00b Joined: 13 Apr 2010 Posts: 38

I'm at the point of trying to mount the per machine directories of diskless clients from a single gentoo image, but having problems for directories such as /var/run, /var/lib/init.d. I do not need to persist the files contents across boots, and am trying to use tmpfs for this directory and have initramfs create the directory layout.

I've installed initramfs using this link ... http://en.gentoo-wiki.com/wiki/Initramfs#Kernel_Configuration ... and the init file in my cpio archive looks like this ...

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

srd · n00b Joined: 13 Apr 2010 Posts: 38

So I think I'm starting to understand a lot of this. I've got a /linuxrc file in my initramfs.img.gz that looks like this …

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

The simplest approach is to quit using an initramfs. Move that /linuxrc into the root of the NFS export, load no initramfs at all, and use the init= parameter on the kernel command line to run your linuxrc before the main init.

As an alternative, you could mount the NFS root as the first step in your /linuxrc, then do the statements you showed, but with the paths rooted at the NFS root. For example:

Rexilion · Veteran Joined: 17 Mar 2009 Posts: 1044

A long shot but ehm:

Can't you just use unionfs from mm?

I mean, create multiple overlay's of the same base system with minor modifications on another FS for each node? And then share each directory? This also allows you to mutate the main image without affecting other systems.

srd · n00b Joined: 13 Apr 2010 Posts: 38

I checked into it, and I could change out my kernel to work w/ one for unionfs, but I thought it better that if I can understand what's going on at the low level, then maybe one day I'll check into unionfs once I understand all this as I'm sure unionfs probably abstracts a lot of the same.