Best FS for data integrity?

py-ro · Veteran Joined: 24 Sep 2002 Posts: 1734 Location: Velbert

Just some Numbers thrown in.

I run a btrfs scrub yesterday.

Processor was a Phenom T1090.

The Array delivered with around 850MB/s, which is all the PCIe 4x Controller could deliver.

This occupied half of the cores and this processor has no usefull extension for calculating the checksums. With a more modern Processor and normal Workload there shouldn't be any problem.

Bye
Py

steveL · Posted: Sat Mar 21, 2015 12:09 pm Post subject:

John R. Graham · Posted: Sat Mar 21, 2015 5:16 pm Post subject:

Edit: Upon further reflection, I've reworded my response somewhat, regarding the applicability of the word, "dismissed".

First of all, I don't really see how that little play on words is categorically different from your suggestion that I might be one of those fellows that insists on wandering around utilizing three different methods of holding up my pants. Friendly banter, both of them, in my opinion. Especially since I did attempt to address your points: you were presenting three different cases that all share a common attribute and I chose one exemplar for comparison purposes, just as I have said "ZFS" when I meant "ZFS or BTRFS". This is pretty standard. But, since they shared this common attribute, they were answered by a common argument. Thus, to say that I dismissed the other two is, I think, accurate.

Second, what I thought you meant by saying that the solution I was espousing wasn't generic enough was that its proper implementation would be at the VFS layer (a higher layer of the OS), where it could universally benefit all filesystems, as opposed to the filesystem layer (a lower layer of the OS) where it is currently implemented, only benefiting users of ZFS (or BTRFS). In that context, I'm sure you realize that my use of "up" and "down" was correct (deliberate, too, but that's not important right now). As I've apparently misunderstood what you meant, I hope you'll explain.

Regarding the parity calculation benchmark, I've finished conceptualizing it, although implementation will take a little longer as I haven't done a serious amount of x86 assembler since the mid-90's. Would it surprise you to know that, using a 5-volume RAID 5 array as an example, including loop overhead, calculating the error correction syndrome uses less than 1.17 machine instructions per 64 bits of I/O channel traffic? The loop where the syndrome is calculated will only contain 6 machine instructions and each iteration of the loop will process 320 bits of I/O channel traffic. Additionally, the SSE and AVX instruction set extensions appear to offer even higher performance than that, but I haven't finished learning about them.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

steveL · Posted: Mon Mar 23, 2015 3:50 pm Post subject:

If you had dismissed my statements earlier, you would have had to refute them first; you made no attempt til the previous post.

As to the substantive, I've lost any sense of motivation to expound further; I don't feel I need to in any case, since krinn got the point immediately. Thus afaic any fellow practitioner has enough info to see what I mean.

In the spirit of the forums, I'll simply add: intrusion detection, wrt "git only saves text" (which is ofc untrue.)

John R. Graham · Posted: Mon Mar 23, 2015 4:50 pm Post subject:

Regarding "dismissed", I don't think that word means what you think it means.

Regarding Git, of course it's untrue, but you were arguing that the human element was important, and one of the reasons why the extra robustness was superfluous at the filesystem level:

The_Great_Sephiroth · Posted: Thu Mar 26, 2015 1:12 pm Post subject:

Wow, lots of good info in this thread now! I actually got incremental working through a home-brew bash script. I would love something like Toucan for Linux, but shell is OK also, since it means I can set it up as a cron job on a server if I had to.

Now I am curious about this ext3 versus ext4 argument brought up. I have not read the entire thread you linked yet, but you are saying ext3 is faster than ext4? Part of me believe that would be normal, since ext4 can handle larger volumes and such, but I am unclear here. I have been using ext4 for five or six years now and have not had any issues with it, even when I have had power issues and a system dies instead of being shutdown correctly.

So this has brought up another question in the back of my mind. What file-system would YOU (any of you) use for a Linux workstation? How about a server? I have been using primarily ext4, but am open to opinion and advice. Now I am going to go read the remainder of the thread you linked.
_________________
Ever picture systemd as what runs "The Borg"?

steveL · Posted: Thu Mar 26, 2015 2:13 pm Post subject:

The_Great_Sephiroth · Posted: Thu Mar 26, 2015 3:57 pm Post subject:

After reading that thread and doing more searching and reading, I came across this performance comparison which showed me that in all of the tests ext4 is fairly good. Overall I'd say it is the best. It even beats ext3 except in one test. I believe I will stick to ext4 except on RAID or Flash.
_________________
Ever picture systemd as what runs "The Borg"?