The Filesystem choice thread

[kernelOfTruth]

[energyman76b]

and how fast would be a tweaked reiserfs?
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.

[kernelOfTruth]

[energyman76b]

[kernelOfTruth]

just found an interesting article on reiser4 and how it functions "under the hood", unfortunately it's only available in German:

http://www.linux-magazin.de/layout/set/print/content/view/full/4459

you can use google-translate to read it in english:

translated article: Fourth Symphony by Reiser

this should explain why reiser4 was developed newly from the ground and is superior to reiserfs v3.6 (especially why it's much more silent during it's file-operations on harddrives)

xattr + quotas plugins development FTW !

we really need some dudes programming the xattr + quotas plugins, cleaning up the code a little and programming the repacker, then it should be ready for mainline-inclusion (... yeah, this won't happen so fast ... )
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004

[energyman76b]

[kernelOfTruth]

[neuron]

Just tried a different kind of test, I copied a 1gb file to a tmpfs volume, thinking I'd copy that to the device in one test, and then copy it from the device to the device.

Just having that mounted (it turned out to 800mb, I got 2gb memory in this computer right now) dropped my performance on both xfs and ext3 quite severly.

Simply unmounting the tmpfs brings the performance back up again. I do "sync; echo 3 > /proc/sys/vm/drop_caches;" between each test, and I can see differences between tests, is a simple dd really that memory hungry? Whats going on here?

xfs from:
WRITE : 285 MB/s
READ1 : 281 MB/s
READ2 : 278 MB/s
to:
WRITE : 238 MB/s
READ1 : 146 MB/s
READ2 : 161 MB/s


ext3 from:
WRITE : 205 MB/s
READ1 : 262 MB/s
READ2 : 262 MB/s
to:
WRITE : 176 MB/s
READ1 : 158 MB/s
READ2 : 164 MB/s

[energyman76b]

seems that your 'test' doesn't make sure that the stuff hits the platter and (partially) tests buffer performance
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.

[neuron]

[neuron]

Couldn't go with an agcount of 4, because it misaligned something on the real raid, went with default, which was 32 (after some small tests, but I couldn't do very accurate tests, because I had 4 raid disks in raid5 with a missing parity drive), after taking the backup off the missing drive, bonnie dropped from 270mb/sec read/write to 160mb/sec write, 215mb/sec read

I did my latest bonnie tests on the testing raid booting with mem=512, and 2gb test files, so it should have been very accurate. No clue why the real thing is so much slower :/, I'd expect some change in performance, but I was hoping for statistical noise only.

When I checked my logs, going from agcount=4 to agcount=16 dropped performance by around 7-8mb/sec on both read/write, but latency went down considerably. I dont think I should be seing a drop this big :/

//edit, ARGH, something is seriously wrong here. Doing copies now I see iowait at around 2-3%, a ton of kcryptd processes, all around 10%, and rsync on around 20%. Nothing is even close to maxing out io or cpu, yet the transfer speed is crap :/

[neuron]

A friend of mine has offered to loan me enough harddrives to backup the full volume, so I figure now I'll do some testing again, and this time I'll do it right.

Gonna run bonnie overnight, booting a kernel with mem=2-300m, crond disabled etc. Also gonna get some graphs done, not entirely sure how, as I'll be doing a LOT of tests, I'll probably split it up based on md chunksize/encryption etc.

mkfs:
I'll do tests with these options, everything will always be mounted noatime,nodiratime:
ext3 stock
ext3 -b 4096
xfs stock
xfs -l internal,size=128m -d agcount=2 4 8 16 32 and 40 atleast
xfs -l internal,size=128m /dev/md2 - For automatic agcount
xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 with sunit=X for logging parameter, to stripe the log (although I'm not entirely sure how that works yet)
reiserfs stock
jfs stock
reiser4 stock
reiser4 -o create=ccreg40,compress=lzo1

mounts:
ext3 mounts will be done with:
noatime,nodiratime,commit=60
noatime,nodiratime

xfs mounts will be done with:
noatime,nodiratime,logbufs=8,logbsize=256k
noatime,nodiratime,logbufs=8
noatime,nodiratime

Raid:
Will vary chunksize 64,128,1024

Encryption:
Will always have twofish-lrw-benbi:sha256 256bit key under the raid, building the raid on top of it, it should be the fastest cipher. Will also do tests with RAW access, ie without encryption at all.
echo "8192" > /sys/block/md0/md/stripe_cache_size

IO Schedulers:
Really unsure here, will try with cfq and deadline atleast, probably my pre tweaked anticipatory that I got elsewhere aswell.

I'll also do tests with these kernel patches (if I can get them applied properly to a recent kernel):
http://lwn.net/Articles/249450/
http://lwn.net/Articles/242559/

Bonnie:
I'll probably base the whole thing around bonnie, but use different sizes (one generic size for the full test, but different sizes for the main filesystem test). REALLY unsure what options to run with here, I've never toyed much with bonnie, pretty much just ran with -S<twice my ram>

How to present all this and how to script something to do all the tests overnight is still on my todo list :p, also not entirely sure how to present the data properly. If anyone has any suggestions on additional tests to do, or any suggestions on kernel patches etc, feel free . Due to time issues I will not run every filesystem test on every different raid chunksize, I'll run one batch test first of all different filesystems on the raid configuration I believe will be the fastest, and then I'll run with the best filesystem optimizations on the different chunk sizes most likely.

[kernelOfTruth]

you're doing really an extensive test here, neuron, for the sake of completeness & also on-topic:

could you add reiser4 to this set ? reiser4 default and reiser4 w. cryptcompress plugin (lzo) would be interesting to know, too

that way we'll have some kind of general overview for all filesystems available for linux

the fs-creation options for r4 are:

[neuron]

[dusanc]

[neuron]

[dusanc]

Well I ran onto this recent test:
http://marc.info/?l=linux-kernel&m=121484256609183&w=2
And some directions from fs devel for one way of benchmarking:
1. a machine with >= 512M RAM and "/" formatted with some fs
supported by old kernels.
2. a spare partition.
3. enable ramfs (it seems it is enabled by default in most distros).
4. put a tarball-to-copy in some working directory (I had
linux-2.6.9.tar.gz)

Note, that some old -mm kernels are not compilable/bootable
(if so, pull the "hotfixes" patches from akpm's directory on kernel.org)

5. edit the attached patches in accordance with your configurations
6. build and boot the testing kernel with reiser4 debug disabled
(I think no needs to boot in single mode, or discard kde, etc..)
7. run "vmstat 2"
8. run ./prepare_copy.sh && ./ncopy on another console

And here's the files:
prepare_copy.sh

[neuron]

Hm, I want cpu stats aswell, so a copy test like that is pretty much out (and it might be a bit tricky to automate), but something that fills the partititon.... that's a good point, I can't do that on the main test, as the volume will be 2tb in size, and if I make smaller partititons to test, my data on stripe sizes and such will be invalid. I might end up with 2 tests, one testing a load of filesystems at high filesystem usage, and one testing raid sizes and such.

And phoronix-test-suite isn't a benchmark per say, which is why I didn't think about it, it's based around using benchmarks and graphing the resulting data, which I want to do, so I'll look into using it to graph my data, or atleast to see what frameworks it uses for it, you might have saved me some work there

[energyman76b]

emm,neuron, for fair results you should turn barriers on for all fs or turn them off.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.

[neuron]

[energyman76b]

[neuron]

[dusanc]

Here's some more recent nice benchmarks http://tservice.net.ru/~s0mbre/old/?section=projects&item=fs_contest2
dbench, iozone, postmark etc.

[energyman76b]

except he 'optimized' xfs some, reiserfs/reiser4 not at all and got extX away without barriers - while all other fs use barriers.

So the results are in favour of extX. You can reduce the performance of them by 30%.

Also, instead of the 'reiser4 was taken from mm' crap he should have used the correct reiser4 patch that you can find on kernel.org.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.

[dusanc]