ZFS ... some sizable elephant in the room?

khayyam · Last edited by khayyam on Wed May 24, 2017 7:03 pm; edited 1 time in total

I've been doing some fairly in depth research on ZFS (in a broad sense, so both "on linux", and the other offerings, such as FreeBSD's port, FreeNAS, NAS4Free, Illumos, etc). I say "in depth" because it probably amounts to about six weeks of reading, viewing, and poking about, as all the while there was something (the elephant, in other words) which kept nagging in the back of my head as each presentation of "the facts" were given, something which none of the "facts" seem to address (hence it's elephantine status).

Without doubt ZFS's feature set is rich, its a veritable smorgasbord whatever checklist you happen to be using to evaluate said "features", so much so that its easy to be swept aloft with the praise ... and awe ... it invokes in those using, and evaluating, it, and by this I mean also the praise, and awe, I too brought to the party. At one point I felt genuinely troubled that I wasn't able to let go and suck at the teat of the great mother of goodness, something didn't feel right, and in my near ecstasy that cup was poison, but delightful to indulge. I wax lyrical, but really, it wasn't something far from the above description ... and I wouldn't know how else to describe it. Perhaps it hit a soft spot, and being generally critically minded it exposed a conflict ... well, anyone might speculate.

So, where's this damned elephant you ask? ... let me propose a thought experiment:

Let's imagine we have a time machine and we could zip back in time to circa 197ish and pander our hindsight to those early boffins of yore writing UFS, or FFS. Let's offer them a filesystem matching all the features provided by ZFS on two reels of magnetic tape, we have the advantage of predating Sun and so the code comes with no licensing issues, it's theirs, and (again, with the advantage of hindsight), neatly plugs into their codebase. Every aspect of the design is explained, and every question re the sort of problems it addresses given suitable reasoning ... there is only one caveat, it's going to cost them 10Mb of RAM (preferably, error correcting ;).

The immediate objection someone might pose to this thought experiment is that our theoretical coders couldn't possible solve software problems without similarly solving the hardware issues that run parallel to them. Yes, but that isn't the point of the experiment, the point is what is the cost of a robust, well designed, feature full, filesystem. Even if there were the hardware to provide required RAM you would still expect them to question the design/engineering given the ratio of resource use in relation to their own terms of reference (writing code with a vastly reduced scale of available resources).

So, as to ZFS, well, should we accept that for the necessitated features (or, in other words, a robust, feature full, filesystem) the requirements (be those, suggested, recommended, or advisable) are reasonable, and match what we might expect the cost to be (given that "ram is cheap", hardisks, SSD's, are similarly, ECC RAM is not much more than non-ECC RAM, etc, etc).

What I think surprised me is that it's not uncommon that someone will say, "8G RAM minimum, but 16G or 32G is advisable", or some checklist listing a line conditioner, UPS, WD Reds, ECC RAM, SSD for ZiL, and another SSD for ARC, etc, etc. I'm not exaggerating, often doing it with less is presented as risky, and I encountered scores of people building such "rigs" for glorified plex servers. Added to this it seems (at least to me) that some liberties are being taken with how such technical solutions are being obfuscated, and the "requirements" raised, so as to bundle VM's (to run docker!!!) into the mix, and necessarily require the additional umpf.

That is perhaps to be expected, but to draw back to the crux of the problem I'm attempting (however awkwardly) to underscore ... what is the cost (not simply economic cost, but what resources should we expect to expend to resolve the problem technically). I now have an AMD64 X2, with 1G RAM, it's a piece of crap I picked up from the garbage, but it runs Dragonfly BSD (with the HAMMER filesystem) seemingly without crapping out due to the lack of resources. HAMMER has many of the features you might expect of a robust filesystem (some, if not all, I'd expect to see replicated elsewhere), and while I would be stupid to claim feature parity, or to make a trivial comparison between it and ZFS, it does beg the question what is ZFS doing that then sets its resource requirements so far above a lesser implementation (if we were inclined to regard it as in fact "lesser"). I'm going to wish I didn't make that argument, because it's obvious we could roll out compression, dedup, zpools, caching, and fill that gap, but however weak the argument I'm still inclined to question why, say, caching, simply doesn't operate with what is available, or how much compression actually costs (by all accounts l4z very little CPU wise), or where the resources are actually used, and for what. I guess the argument could be made that at least some of the resource usage could be the result of lack of optimisation, and indeed, there is some truth to that (the obvious case is doubling with regard to ZFS cache, and the VM) but I still think it would be difficult to explain everything it terms of optimisation, or features, alone. That explanation, or at least something approximating it, is the elephant in the room, and the problem I have with drawing the two ends of the spectrum posed in this missive together, yes, ZFS is that mythical beast of a robust filesystem, but like the beasts of yore, it may need to be placated by depositing the daughters of the kingdom at its feet ... thoughts?

best ... khay

edit to add: forgot to mention, as a test of robustness, I created a HAMMER filesystem on a USB stick, and tryed repeatedly mounting and pulling the stick out (without unmounting), dd'ing random blocks, and intentionally trying to damage it content. Pulling the stick made no difference (and no fsck was required subsequently), and all files were easily recoverable just by rolling back (HAMMER keeps revisions automatically).

John R. Graham · Posted: Wed May 24, 2017 6:59 pm Post subject:

I think your elephant may, in fact, be a mouse, or perhaps a vole. My intuition is that most of what you read about those massive requirements is to support massive filesystems, something that ZFS was designed to do. In contrast, a modestly-sized filesystem (by today's standards) probably requires commensurately modest resources.

Edit: That said, I did have stability problems on a 32-bit ZFS installation I tried about three years ago. I'm mostly switched over to 64-bit now and am planning to experiment again. On paper I really like ZFS but my opinion above is not based on much practical experience.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

khayyam · Posted: Wed May 24, 2017 7:22 pm Post subject:

John R. Graham · Posted: Wed May 24, 2017 7:36 pm Post subject:

No, but I'll bet (intuition again, not yet profound knowledge) that the relationship is more logarithmic than linear.

Maybe we can entice ryao into venturing an opinion.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

khayyam · Posted: Wed May 24, 2017 8:26 pm Post subject:

nicop06 · n00b Joined: 25 May 2017 Posts: 13 Location: France

I confirm, ZFS needs at least 4GB of RAM and a 64 bit OS to work if you have normal size disks (>1TB). Let me develop.

I used ZFS on FreeBSD 9 3 years on my NAS server, a 32 bit Pentium 4 with 2GB if RAM and 2 1TB drive. The main feature I used was subvolues, the snapshot system exported via samba, the file level checksuming with regular integrity check using scrub command. I also had a software RAID 1, but this can be done without ZFS.

First, I had to increase the ZFS cache size setting to avoid kernel panic. I recompiled the kernel with KVA_PAGES set to 512, and set the vm.kmem_size_max to about 640M (more made the system unbootable). I still had kernel panic on disk intensive workload, i.e. doing backups via SMB.

So my recommendation is: do not use it on your 1GB system, especially if you use a 32 bit OS, unless it's on a very small disk.

Also, I didn't know HAMMER and it seems to fit almost all of my needs. I am currently using BTRFS but I am not happy about performance. How does HAMMER compare to UFS/ext4 performance wise?

khayyam · Posted: Thu May 25, 2017 11:48 am Post subject:

nicop06 ... thanks for providing those details.

P.Kosunen · Guru Joined: 21 Nov 2005 Posts: 309 Location: Finland

I think biggest problem on ZFS was inability to grow existing raid-z pool with new disks. I used it when it came to FreeBSD current with only 2GB memory. It needed some tuning even then because of low memory and there were no dedup, packing and other fancy features yet available.

Zucca · Posted: Thu May 25, 2017 12:25 pm Post subject:

mrbassie · l33t Joined: 31 May 2013 Posts: 772 Location: over here

Zucca · Posted: Thu May 25, 2017 3:23 pm Post subject:

mrbassie, thanks.

I guess my choice of btrfs was right for me. It sure is slow, but gives some redundancy and ease of use when I replace old (broken/small) drives with new ones.
If I had some 10-20 disk array data storage, then I would propably use ZFS there. At the moment I have 5 and 6 disk setups and I do not plan to grow the number of disks.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

nicop06 · n00b Joined: 25 May 2017 Posts: 13 Location: France

khayyam · Posted: Thu May 25, 2017 10:09 pm Post subject:

nicop06 · n00b Joined: 25 May 2017 Posts: 13 Location: France

Zucca · Posted: Fri May 26, 2017 9:16 am Post subject:

I found this pretty interesting: YouTube - Allan Jude Interview with Wendell - ZFS Talk & More
And it's not too old either.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

khayyam · Posted: Fri May 26, 2017 2:24 pm Post subject:

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

On the topic of resources, to be fair ext4 needs a lot of resources too - when the filesystem is huge, corrupt for some reason or other, and needs fsck it quickly might also demand several gigabytes of RAM. People with large ext4 filesystems on a NAS (small cpu/ram, huge disk) sometimes find themselves unable to recover w/o the help of a bigger machine. If you have a choice, it's better to create several smaller filesystems instead of one dozen-terabyte monster. LVM makes it easy to slice your storage and resize anytime.

khayyam · Posted: Fri May 26, 2017 4:45 pm Post subject:

P.Kosunen · Guru Joined: 21 Nov 2005 Posts: 309 Location: Finland

ShadowHawkBV · Guru Joined: 27 Mar 2004 Posts: 352

I know that the plural of anecdote isn't data, but here I go anyway. My zpool server is also my main work system. I have firefox with at least 17 tabs open, evolution running and pulling data for three e-mail accounts as well as managing my and my better half's calendar, Makemkv ripping Blu-rays or DVDs to disk(the ZFS pool), Handbrake turning the rips into 1080p H.265 mkv files and saving them to the ZFS pool, streaming a mkv files off the zfs drives over CAT 6 cable to be played on the big screen TV my an anemic mac-mini running Mint, serving and receiving other files to the other four computers and two tablets in the house, acting as the proxy server for the Windows box, and doing daily emerges.

I don't consider this system all that beefy. It has 16GB of non-ECC Ram, Phenom II 955 CPU, an antique NVIDIA video card, 2 PCI 4 port SATA 2 cards, and 2 GiGE ethernet ports. All in all it's old, but reliable. My zpool started as 5 1TB drives in RAIDz1, I later added 2 1.5 TB drives. I'm now running 2 4TB, 3 3TB, and 2 2TB drives with a total of 11TB of mirrored and striped redundant storage. These hard-drive upgrades are a combination of drives failing, and upgrading drives with higher capacity ones as I get motivated. (I think I've been running this for 6 years now - this system is only turned off when there is a power-outage that lasts longer then my UPS can handle - 30min or so) None of my drive failures or replacements have caused me to lose data. Me being an idiot, has caused me to lose data, but that's not the computers fault.

bunder · Bodhisattva Joined: 10 Apr 2004 Posts: 5934

I just noticed this thread. I probably won't be able to cover everything at the moment, but I thought I'd throw in my 2 cents since I've been using ZFS for roughly 2 years.

The memory requirements are a fallacy. If you don't want a huge cache, then that's up to you. L2ARC and SLOG aren't necessary for 90% of home setups. If you want to be baller, then go for it. I highly recommend using 0.7.x because of memory usage improvements.

Unfortunately that's all I can write for the moment, feel free to pop into freenode IRC #zfsonlinux, I'm sure some of us would be willing to go on for days about usage, requirements and performance.

edit: Me again. I thought I'd throw in a link about the supposed need for ECC. http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ (short answer: not any more than any other filesystem)