read-only filesystems

[Atom2]

Hallo Forum,
I am in the process of building up a XEN gentoo system. My ideas are currently as follows:

Dom0 with a hardened system - that's basically up and running. I can start PVs and they work.

Next steps would be:
A number of paravirtualized gentoo DomUs for various services like DNS, DHCP, squid, samba, nagios, SMTP, IMAP, etc.
Over and above this I'd plan to use FreeNAS as the storage system - clearly with a HVM and direct access to storage disks (PCI-passthrough).
For the firewall my idea currently is pfsense (FreeBSD), also with HVM and PCI-passthrough of a network card for the internet access.
Lastly a few win7 HVMs.
Logging from the paravirtualized DomUs on gentoo would probably use remote logging to the Dom0 syslog-ng daemon.

In order to minimize the administration of those PV gentoo DomUs my idea is to provide them with identical kernels and also share common files (e.g. binaries). From my understanding this would work provided the relevant filesystems can be mounted r/o.

The same principal of sharing a common system disk image might also be on the plate for the win7 virtual machines, but that's still a bit down the line at the moment.

So my question is: Which filesystems / directories can be mounted r/o without limiting operation of a gentoo system during normal operation.

Updates could work as follows: There are two sets of all those shared files / directories (production & test). Updates to newer versions could be done on the test set and then tested in a separate virtual machine. Once o.k., the changes could be rolled into the produciton set. After that only a restart of all virtuial machines would be required and all VMs would have the latest set of shared files once they are up again.

I'd think that the following directories would need to be separate and mounted read/write:

[vaxbrat]

You might want to look into using aufs to do a union of a base filesystem as readonly and then a custom one for each guest that gets rw mount with whatever files it has/writes unique from other guests. Most of the time, the readonly one is also compressed (ie squashfs) since the original use case for this approach was for thumb drive or network boots. If you rummage around in the wiki, you will see the howto's for thin clients, etc doing this.

[Atom2]

Hi vaxbrat,
that approach sounds interesting indeed. I am not sure yet how that all plays out with XEN, but I assume it requires an initramfs (which, if possible at all, I'd rather avoid).

In any case thanks for sharing your idea; I'll have a closer look, although ressources to read into the subject seem to be scarce. I only found one article specifically related to XEN and aufs here. If you happen to have a few more links available, I'd very much appreciate that.

Another approach that I came across when searching for shared storage under XEN is to use an LVM-backed block device and create a snapshot for every guest as briefly described here. That would do away with the requirement of an initramfs as far as I understand it and - given that I do already use LVM (albeit without snapshots so far) at the moment - might be an easier way forward. If I am not mistaken, the snapshot could be mounted rw - so no special treatment or ro mounting required - and the storage basically behaves like a cow filesystem ...

Any further ideas/suggestions are welcome as well as opinions on which of the two approaches above are preferrable.


Atom2

[vaxbrat]

I've been getting more into using btrfs lately but haven't dived yet off the deep end by doing it as a root filesystem. That has snapshotting, subvolumes and can do compression on the fly. You should be able to avoid an initramfs using it by just specifying the volume label on the kernel line.

[Atom2]

vaxbrat,
after a lot of research and reading, I have decided to try the route of your initial idea with squashfs and aufs.

I have also managed to make some progress and have a pv gentoo guest up and running including an initramfs. There are however a few problems which need ironing out. The most pressing issue is that whenever I shutdown the domU, the two mounted filesystems that make up the union overlay filesystem with aufs refuse to be unmounted. This results in the r/w filesystem part (an LVM volume formatted as ext2 that serves as the overlay to the r/o squashfs) to get corrupted due to unclean shutdown and at every reboot there's a fsck required (which so far always only clears a few inodes). The r/o squashfs, which also refuses to unmount during shutdown, obviuosly doesn't have that problem due to its read-only nature.

I have no idea how I can overcome this as the unmount obviously thinks that these two filesystems are busy (but at the same time states, that fuser can't find any active process using it), and it certainly does not increase my confidence in this setup if I (potentially) face data loss at powerdown. I have also manually tried to unmount them from within the virtual machine, and that also doesn't work; also remounting with r/o access does not work. If I can's solve this, I think unfortunately that's a no-go.

The other issue, albeit much less pressurizing, but still very annoying, is that after the init process from my initramfs makes the switch to the real root and passes control to the standard boot-process of gentoo, no more boot messages show up on the virtual xen console. After a few seconds of silence, however the console login comes up and the system is fully usable. I assume there's someting not in order with device nodes, but haven't been able to track down what it actually is.

The other ideas with snapshots (whether LVm or btrfs) all have one drawback that I either don't understand correctly or make that approach much less appealing: Updates of the original data underneath the snapshot (e.g. in case of new releases of gentoo or additional packages) would not really work as easily as with squashfs. As far as I understand changes to the original filesystem before the snapshot would result in all old files to be propagated to the snapshot resulting in asubstantial increase in its size.

That would not be the case for squashfs and aufs and that's why so far I think aufs is the sexier option of the two. That's mainly because making updates is so much easier: You just need to create a new squashfs and you are ready to go.

That's my current understanding, but who knows, I might be wrong, so all inputs are very much appreciated ...

Thanks and regards Atom2

[vaxbrat]

I looked at xen once a number of years ago when rhel was big on it in early 5.x versions. However I've been with kvm when I have the choice and am stuck with vmware when the windows guys at work stick me with that underneath.

Now that I've gotten a bit more warm and fuzzy with btrfs I've gotten around to deploying ceph on top.