Clustering on Gentoo - What's the best option

[eohrnberger]

Hi!

I have to admit that I'm a bit new to the clustering aspects of Linux, but I've done some reading on the Internet and it's left me a bit confused as to what's the 'right' decision WRT clustering and distributed file systems for Gentoo.

Me? I've been running Gentoo since '02, and I have to admit that I don't think I'll go back to using a binary distro again. Why live with someone else's compromises when you can make your own?

The Home Network:
I have a main storage system on the network that serves as a NAS for Linux as well as the TiVo using Galleon, and the Windows clients. It has about 7 TB of online storage pools, which is using ReiserFS. I've just about managed to squeeze out all the EIDE HDs so it's mostly SATA now. This is where all the files are kept. It serves video, music, software repository and file backup drop for the data files for the Windows desktop clients, etc. This is a 64 bit Intel dual core 3 GHz machine with 4 GB RAM. This system runs well, and since I switched to Gentoo, hasn't lost a single file. Yeah! It's running a 3.10.17 based kernel, and it's using EVMS as an LVM2 management interface (kinda tells you how long it's been running and evolving).

I've also have 2 compute servers (likely to grow in number), both are Intel 64 bit dual cores 3.0 GHz, inherited cast offs, also on the network, also running Gentoo, and presently NFS mounting from the storage / NAS server. They are running a 3.10.25 based kernel.

All this is connected with a Gbit switch and each server is connected with a Gb NIC.

Lately, playing around with VirtualBox, with a variety of guest OSs for a variety of purposes and needs.

Right now, I have a central registry of VMs on storage system (the VirtualBox.xml file), with virtual HDs mostly on the storage machine, but a few on the compute servers (they have smaller LVM2 storage pools attached as well). It'd be nice to be able to run a VM on the least loaded compute server, and access the virtual HD on whichever server, storage or compute it resides. Problem is, when running a VM across an NFS mount, it really crawls.

Reading on the Internet, the reports are that NFS is really slow in high demand / high bandwidth situations, so the thought is that to make the greatest use of the Gbit network, the network file access protocol needs to change. Seems like this means iSCSI targets for the server and iSCSI initiators for the clients. That part emerged nicely.

Since it appears that iSCSI is going to be used as a network transport protocol, and since iSCSI doesn't have a clue about filesystems, only data blocks that need to be read and written, locks, distributed cache coherency, some sort of cluster aware file system needs to be used to keep things in sync and not corrupting any data, hopefully one that is bandwidth fast, but also more towards meager in resource consumption (I don't have the hardware budget to stand up a dedicated server as a file access traffic cop for example).

The eventual goal, which may not be possible, is to start a VM on one compute node, and have the workload migrate to other compute nodes as their demand drops off, sort of self-balancing. These abilities seems to be specialized in the extreme parallel computing, and not for heavy weight VirtualBox VMs.

Since it's a home network, there really isn't a lot of transactional demand (there's no user community of clients banging away at it), it's more like I'm the only real user, and only when I'm running VMs.

I tried emerging GFS, but that's locked to way back kernels with it's need for smp_lock.h (the great big kernel lock - or so I've read). I also tried emerging the gfs2-utils, thinking that it would drag along the needed kernel changes, but that also didn't emerge due to some compiler error.

Yeah, I probably need to update the kernels, and that shouldn't be a big problem, but the first thing to do is to test the cluster setup in a one server, one client pair of VMs, just to get the needed practice in before changing the physical systems. Since I have 64bit Gentoo VMs hosting on Windows machines as distcc servers, I started with one of those as a base, and am updating it as a starting substrate for the test VMs.

So we got the direction and the end goal. We have the usage pattern, and we have the first steps to testing the setup. What's left is which cluster software and possibly which file system to use (although if at all possible, I'd like to keep the filesystems that are in place now - but hey! I've been meaning to get an 8 port SATA controlled for the PCIe slot on the storage machine, and a set of new 2 TB hard disks which to LVM2, so if needed I can start working on spreading the purchases out a bit and converting (although I have to admit, I'm a bit concerned about concentrating the IO demand across so few spindles and across a single bus).

So I ask the more knowledgeable people of the forum, what's the right direction / choices to make here for continuing the evolution of my Gentoo network? What's easy to administer, reliable (as most Linux software is) to meet this need?

[szatox]

[eohrnberger]

[szatox]

[eohrnberger]

[eohrnberger]

Well now. Interesting development.

I can in fact run a VM on a compute server and have the virtual hard disk file on a remote server across an NFS mount. The performance impact is a little bit noticeable, but well within the acceptable range, at least to me.

The bit of magic that if ferreted out the Internet was to apply the following options to the NFS mount options:

rsize=32768,wsize=32768,hard,bg,intr,retry=1000 0 0

This increases the read / write blocks to 32 KB, allows a long interruptible time out should the NFS server be off line for a period of time.

I think I'm still going to explore Ceph, as a means for distributing disk space across multiple machines, and concatenate it all together into a single larger storage object as presented to the clients.

As we concluded, a NUMA type architecture using Gb Ethernet as a communications medium for process migration just isn't going to work, so giving up on that for now.

Thanks for being a sounding board on this. Much appreciated.

[szatox]

ok, 3 more hints about NFS:
in /etc/exports add "async" option - this in not completly safe, as any writes to NFS share will report success BEFORE actually writting anything. But it will boost performance and is a good pick for non-critical stuff.

Try using jumbo frames on your network: bump MTU from default 1500 to 9000. Some network devices don't support it, but if they do it reduces protocol overhead a bit.

Check if you have enough nfs daemons. I don't remember how it was done, but if you don't have enough servers to handle the load it will generate some errors you can track. General suggestions were "at least 1 per core, but 4 or 8 per core might be a better start". I doubt I would need this many, but you want to put some strain on those so your mileage might vary.
# Number of servers to be started up by default
OPTS_RPC_NFSD="8"

[eohrnberger]

[eohrnberger]

Well, it's been a busy couple of weeks here. Thought that I'd better pot up and bring everyone up to speed.

I now have ZFS implemented on all 3 of the dual core x64 PCs, which includes the storage system (NAS). All EIDE drives have been squeezed out and replaced with SATA 6.0 Gbps drives (well, there were only 2 left).

A new PCIe 4 port SATA controlled added, and the 6.0 drives plugged in (too bad both pools cover a mix of 6.0, 3.0 drives), but now identified, going to start squeezing those out and replacing them with the 6.0 drives as well.

ssh is now password-less from any machine to any machine, as are all the NFS / Samba shared. There's even 2 more x64 VM guests on Windows hosts that have been integrated to the same level. All have been added to the local DNS server, so going from anywhere to anywhere works seamlessly, as it should.

The 2 file systems on the NAS system have both been converted to ZFS, and the vast majority of the files have been COW re-distributed across the drives in each of the storage pools. The 2 storage pools (primary and secondary) are made of discrete sets of hard disks, 8 each, so if a drive up and dies, it'll only impact a single storage pool.

zpool iostat says that highest throughput was around 280 MB/sec, which I guess is pretty good striping across the drives considering the mix of high and low performers in the group. Interesting to note that in cases of heavy IO, from local pool to local pool, the system load climbs pretty quickly to over 12. Rather worry some. Better if under these loads, processes were pushed off of the storage machine, and onto the other machines, to relieve that overburden, and then migrate back, after the IO load subsides.

As indicated earlier, running VMs across the NFS mount is now possible, at a slight, but acceptable, performance penalty.

Now, it's back to thinking about which SSI clustering to implement. Are there any in portage? (looked but haven't found any). It's been a joy finding all the rest of the experimentation, zfs, btrfs, ceph, etc. and all the rest in portage, just waiting to be emerged. Kudos to the portage guys!

[szatox]

[eohrnberger]