Correct NBD (Network Block Device) usage?

Message

jjlawren · Post by **jjlawren** » Tue Jun 27, 2006 3:01 pm

I'm having issues using NBD devices correctly.

1) For a diskless machine (using NFS for / and NBD for larger directories such as /usr and /var), what is the best way to start nbd-client sessions before the partitions are mounted?
2) If NBD is used for sharing a portage cache as a read-only squashfs filesystem, how can the cache be updated? Overwriting the squashed file system and simply remounting the filesystems on the clients keeps showing the old cache. Does the NBD device need to be disconnected at the kernel level and then recreated?
3) What's the best way to stop an NBD share? I can't seem to kill the nbd-client or nbd-server processes without using a kill -9. Unloading the module gives an 'in use' error.

Paladine01 · Post by **Paladine01** » Tue Jun 27, 2006 3:25 pm

1) You have to patch the kernel to include an NBD client. This has been done for Gamecube Linux. Check out www.gc-linux.org. You'll find some information there. isobel is the guy you want to contact.
I HIGHLY suggest you stick with NFS. It just *works* and you don't have to fiddle with NBD. If you're mixing the two anyways, why not just stick to NFS everywhere?
What you could do is mount your / directory over NFS and include the nbd-client in this filesystem. Then use the nbd-client to start your other mounts. You'll have to use a script to do it, fstab won't cut it since you'll need to manually execute your nbd-clients.
2) Not sure. I avoid NBD like the plague. In fact, I only use it for swap.
3) You have to unmount the filesystem first, then kill nbd-client. As long as its unmounted, kill -9 should have no ill effect.

jjlawren · Post by **jjlawren** » Tue Jun 27, 2006 3:49 pm

I'm trying to move away from NFS since it seems like NBD performance is much better.

As for 3) I'd rather not have to kill -9 every time. And it seems that when I do this, the server side sometimes starts to use 100% CPU time.

Paladine01 · Post by **Paladine01** » Tue Jun 27, 2006 4:42 pm

NFS performance can be greatly tweaked.

Are you using nfsvers=3?

How about rsize,wsize?

If you're behind a LAN with an MTU of 1500, you'll want to set rsize=1024,wsize=1024 in order to fit your packets inside one frame. nfsvers=3 is also critical. The default is 2.

Give those a shot and see if it performs better.

jjlawren · Post by **jjlawren** » Tue Jun 27, 2006 5:08 pm

I'm currently using the following mount options:

noatime,rsize=32768,wsize=32768,nfsvers=3,tcp,actimeo=60

The transfer speeds are much better when I set my MTU to 9000, but that effectively kills my internet connection (router can't seem to handle them). Perhaps the r/wsizes are too large for 1500. I'll try making them smaller. Is 1024 really the best size--I've read so many places that rsize=wsize=8192 will increase performance greatly. Now that I actually think about it that doesn't make much sense.

Edit: Just ran some tests, and rsize & wsize settings are much, much better with larger values. I'll stick with 32768.

Paladine01 · Post by **Paladine01** » Wed Jun 28, 2006 9:32 pm

Wow, those sizes are huge. And TCP is generally not used, UDP is the typical way to go. Are you running this remotely over the Internet? The suggestions I proposed were meant for a local LAN.

Either way, trial and error to find the settings that work best for you.