Btrfs nas?

The_Great_Sephiroth · Posted: Sat Jun 18, 2016 12:51 am Post subject: Btrfs nas?

Before mentioning Rockstor, don't. I like Rockstor, but due to some oddball design where it sees entire disks as block devices which cannot be partitioned, you need three disks for a RAID1 (OS, then the array). That is plain dumb and wasteful. As such, I want to build my OWN NAS using Gentoo and BTRFS.

I am thinking of doing a BTRFS RAID10 array using four 500GB disks I have laying around. This would give me 1TB of space with increased performance. My goal is to install Gentoo in as small a configuration as possible. It will be shell-only, no Apache or anything, Samba for being a domain member and hosting shares to my gaming rigs, smartmontools on a cron job to monitor disk health, and probably cron jobs for maintaining BTRFS.

I want to control disk spin-downs. I really don't want them to spin down. Spin-downs kill desktop disks much faster than laptop disks, and if I go on vacation, I'll just shut it down myself!

So what advice can you give me before I begin this project? The OS partitions will be on BTRFS also, in case a drive dies the OS can still function. Oh and that reminds me, how about hot-swap in the event of a failed disk?
_________________
Ever picture systemd as what runs "The Borg"?

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

I would do the same thing myself if I only had one box. Instead I use 6 in a ceph cluster, but I'm based on btrfs mirror sets for my object stores. If you want to consider this, you would want a minimum of three boxes, two for monitors and object stores, and a single one as the quorum monitor, metadata server and the host for sharing samba and nfs.

Based on your comments, I assuming you intend to put your system boot and root on the same 4 drives as the mirror set. Use btrfs as your system root along with grub2 (or other boot loader that is btfs savvy) and a kernel that has btrfs support built natively and not as a module. I did a writeup on the wiki a few years back about btrfs system mirrors that may still be useful:

https://wiki.gentoo.org/wiki/Btrfs/Native_System_Root_Guide

Make sure you enable raid10 for both metadata and data. Also use a label to make things more readable in the fstab:

The_Great_Sephiroth · Posted: Tue Jun 21, 2016 6:35 pm Post subject:

I would use UUID's, but I am not familiar with snapshots yet. I did some basic reading on them, but nothing major. I do intend on learning them and using them though. I WILL be using an eSATA disk as backup as well.

You gave me a BUNCH of info and I believe that I am ready to move on this. Step one however, is to create my new AD DC using Gentoo, then move on to the snapshots and backup. I am at work and have to keep this quick, but I wanted to thank you for your informative reply.
_________________
Ever picture systemd as what runs "The Borg"?

alexcortes · Posted: Tue Jun 21, 2016 9:20 pm Post subject:

Unless the objective IS to use BTRFS I would go to ZFS instead. Also, for NAS installations I usually prefer to have the system on a separated small disk, ever on a flash device.

https://wiki.gentoo.org/wiki/ZFS

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

I think the telling thing about this is that while Oracle owns the IP for both zfs and btrfs (inheirited from the Sun buyout), they use btrfs for their dbms backend on their bastardized version of RHEL. Also Facebook is a major proponent of btrfs. I've personally been using it for about 4 years now either as standalone raids or as the underlying filesystem to ceph (for about 2 years since Firefly). The only rare hiccups I've ever had with it were do to bad memory on consumer hardware (ECC would fix that) or from hard lockups due to hardware or power failure. In those situations I've been able to copy just about everything off of the bad filesystem since it mounts readonly for all but the most screwed up of cases. When you run a clustered filesystem like ceph on top, you just drop the Object Store Daemon (OSD) that runs the bad filesystem, re-intialize and then re-add the OSD. The cluster does all of the repair work necessary in the background to rebuild the replicas.

I might consider using zfs if I were on a bsd setup, but btrfs has a few more advantages including the ability to both grow and shrink the volume pool on the fly.

The_Great_Sephiroth · Posted: Wed Jun 22, 2016 7:05 pm Post subject:

ZFS requires too much overhead. Loads of RAM, for example. I like BTRFS and use it on a few laptops here that run Gentoo. We use it on the home partition with zlib compression (save more space since data is duplicated) and it works fine.
_________________
Ever picture systemd as what runs "The Borg"?

alexcortes · Posted: Wed Jun 22, 2016 9:58 pm Post subject:

Just to point out: ZFS uses a lot of ram just if you use dedupe.

The_Great_Sephiroth · Posted: Wed Jun 22, 2016 10:11 pm Post subject:

Yes, and BTRFS supports deduplication also, but at much less cost from what I have read. I believe that combining deduplication with zlib compression should offer a lot of space saving on a RAID array used for file storage.
_________________
Ever picture systemd as what runs "The Borg"?

The_Great_Sephiroth · Posted: Thu Jun 23, 2016 2:17 am Post subject:

Vaxbrat, I am new to CoW on this level. I have used systems with CoW before, but never configured or fully understood it. Can you help point me in the right direction here? I want to understand how it works and how to set it up. I have read reports that using CoW and compression results in less compression or no compression.
_________________
Ever picture systemd as what runs "The Borg"?

brownandsticky · n00b Joined: 19 Nov 2014 Posts: 4

If you suspect the effort and cost will get too high. I'll give a nod to Netgear RN100 series of NAS's.
They use BTRFS on top of md RAID.
A dual bay' is serving well; initially as a NAS and now as shared storage for Xenserver. Admittedly the throughput limits it's usefulness as a Storage Repository.

The_Great_Sephiroth · Posted: Fri Jul 01, 2016 1:08 am Post subject:

I just came upon a question. How do I setup fstab with this? Each partition has a unique UUID and the RAID has a UUID. For example, sda2 and sdb2 have UUID's, but when formatted with BTRFS in RAID1, the RAID1 virtual device has a UUID. Can I use the RAID UUID in fstab? If not, what happens if I use the UUID for sda2 and the entire disk fails? Does it know to mount sdb2 instead? Does it not mount at all and crap itself? What?
_________________
Ever picture systemd as what runs "The Borg"?

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

You probably want to use the UUID that you get back from btrfs fi show:

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

It's safe to turn on compression and not worry whether it works out or not. When writing an extent, the btrfs worker runs the compressor on the buffer before sending it out. If it compresses down, the result is written out. If the result is the same or larger, the original buffer is simply stored. Basically its the same approach taken if you try to run 7zip, jar or the like on a directory with contents that's already compressed such as jpg images or compressed video.

So the resulting filesystem size really depends on what you are putting on it. Resulting i/o performance may be substantially faster than expected if the compression is on the order of 2:1, or more or it will be more on par with whatever bandwidth you have on the hardware. For a rule of thumb, the sequential write performance of a spinning disk pretty much maxes out at 100mb/sec.

Because my cluster spreads out its I/O, it's not uncommon for me to see copies of 10gb video files going out to the object stores hitting over 200mb/sec even when they don't compress well, but that's what you get when you run with the big kids

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Sorry for the interruption, I'll keep it brief...

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs