FS Benchmarks: comparing Reiser, XFS, JFS, ext3, ext2

[platyk]

So I'm somewhat new to Gentoo and I couldn't find any clear references to give me an idea which filesystem I want to install Gentoo on: ext3, ReiserFS, the mysterious XFS, or the even more mysterious JFS? So I decided to create a little benchmark to compare them all, and I'd like to share my results here. The differences between these FS were dramatic.

The benchmark focuses on common operations on a big tar file which contains lots of little files: cp, tar -x, tar -c, chmod -R, ls -R, rm (more detailed description is at the bottom). All operations are performed sequentially, single-threaded. So these results apply more to simple desktop use, and less to multi-threaded server type stuff.

Results:
Times are in seconds. Blocks Used shows the peak amount of space used as reported by df -k.

Code: Select all

File                                 Real     User   System  Total 1K  Blocks
System    Mount Options              Time     CPU    CPU     Blocks    Used	
--------  -------------------------  -------  -----  ------  --------  -------
ReiserFS                             282.414  6.050  66.980  15719084  1584988
ReiserFS  -o noatime                 256.560  6.250  66.740  15719084  1584988
ReiserFS  -o notail                  178.170  5.760  71.330  15719084  1817160
ReiserFS  -o noatime,notail          173.601  5.830  70.950  15719084  1817160
XFS                                  528.737  6.870  57.550  15714768  1806612
XFS       -o noatime                 478.343  6.800  56.520  15714768  1806612
JFS                                  516.397  5.860  31.190  15686116  1860644
JFS       -o noatime                 482.550  6.270  30.100  15686116  1860648
ext3      -o data=journal            719.786  5.740  34.980  15472768  1870304
ext3      -o data=ordered            713.118  6.090  33.760  15472768  1870304
ext3      -o data=writeback          782.316  6.120  34.010  15472768  1870304
ext3      -o data=journal,noatime    635.717  6.020  34.750  15472768  1870304
ext3      -o data=ordered,noatime    633.591  5.940  34.120  15472768  1870304
ext3      -o data=writeback,noatime  710.749  6.300  32.660  15472768  1870304
ext2                                 707.005  5.480  23.570  15472768  1837496
ext2      -o noatime                 652.535  5.760  23.120  15472768  1837496

Conclusions:
ReiserFS is by far the fastest under this benchmark. It is even faster with the notail option. ReiserFS notail is a whopping FOUR times faster than ext3 data=ordered!!! Without notail ReiserFS is considerably more space efficient than all other FS. With notail its space efficiency is similar to all other FS. However ReiserFS burns considerably more System CPU time than any other FS to perform the same operations, so I speculate that its performance may not be so great in situations with a heavy CPU load.

All of the FS get a little speed boost from the noatime option.

XFS is unremarkable. JFS is interesting in that it has very low CPU use and achieves a speed faster than ext2/ext3 (but not as fast as ReiserFS). Maybe JFS would be a good choice for a system with a slow or heavily loaded CPU?

For ext3, the data=ordered option is fastest and the data=writeback option is the slowest. This contradicts the mount man page which says data=writeback "is rumoured to be the highest-throughput option".

Benchmark system:
Motherboard: Intel D815EEA
CPU: Intel Pentium III 733 MHz
Drive: Western Digital WD1600JB

Benchmark kernel: gs-sources-2.4.21_rc8

Benchmark file: mozilla-source-1.3.tar.bz2 from /usr/portage/distfiles
Time to uncompress with bunzip2 is not counted in the benchmark since this is CPU bound.

Benchmark operations:
mount
cp mozilla-source-1.3.tar copy1.tar
mkdir out1
tar -x -f copy1.tar -C out1
cp -R out1 out2 # Slowest operation on all FS, so this gets the most weight.
tar -c -f copy2.tar out2
mkdir out3
tar -x -f copy2.tar -C out3
chmod -R uo-rwx out2
ls -R out3 > dev/null
ls -lFR out1 out2 > /dev/null
df -k --sync
rm -rf out1 out2 out3 copy1.tar copy2.tar
umount

[gmichels]

Hey nice work, I always wanted to know the differences between the filesystems.

I use reiserfs mainly because the install guide says it's stable and fast, but I never had facts.

About the cpu usage, maybe reiserfs4 will have improvements in this?

[sphyko]

Ah, excellent, I've actually thought about trying to find one of these myself. Thanks for sharing, I know Reiser had to be the best

[scrooch]

Hey nice work, I always wanted to know the differences between the filesystems.

I use reiserfs mainly because the install guide says it's stable and fast, but I never had facts.

About the cpu usage, maybe reiserfs4 will have improvements in this?

I think it will be some worse with reiserfs4, because reiserfs4 will use a database which requires some cpu to maintainance.

[Serum]

Very interesting, thanks for sharing your findings. I'll bookmark this thread for later use.

[i3839]

Nice benchmark.

Some remarks:

It seems that you used different partitions for the filesystems, that could mean that the HD speed is different too, influencing the results a little. Fairest would be to use one partition for all the testing, just format the partition after boot and run the test, to make sure nothing is cached and that they all have the same circumstances. Maybe giving the result of hdparm -t /dev/hdX is interesting.

If "cp mozilla-source-1.3.tar copy1.tar" is from /usr/portage/distfiles then you also test the FS on the portage partition, possibly giving an advantage to the FS that is the same as from where you copy the file. Copying the file will cache it too. I think it's best to exclude this step in the benchmark if the file is on another partition.

If Reiserfs is about twice as fast as the other FS's, then it's no wonder that it also uses more CPU, it just does more in the same time. I don't know how you measured the CPU exactly, I assume it's the average CPU usage while running the test. If it's the average CPU usage then (user CPU + system CPU) * Real Time is interesting too, because that's the total CPU usage used for the test.

[Anacific]

Nice job!

[nsahoo]

Thanks platyk for inspiring me. As I don't understand most of the published benchmarks I decided to test it myself. I compared ext3 mounted with noatime and reiserfs(3.6) mounted with noatime, as I was planning to move from the ext3 to reiserfs.

I nuked my redhat partition after about an year and formated it as reiserfs (3.6) and installed gentoo. my currently working gentoo is on ext3. some information about my system.

Code: Select all

bash-2.05b# uname -a
Linux nachi 2.4.22-gentoo-r1 #1 SMP Tue Dec 9 15:16:22 Local time zone must be set--see zic m i686 Intel(R) Pentium(R) 4 CPU 2.80GHz GenuineIntel GNU/Linux

It has 512MB DDR. Both reiserFS and ext3 partitions are on a WD 120GB HDD running at 7200rpm.

Below are the results.

data

Code: Select all

bash-2.05b$ ls -R bin_store/ | wc
  40675   40756 2305001
bash-2.05b$ du -sh bin_store/
623M    bin_store

bash-2.05b$ ls -R data | wc
    713     712    9695
bash-2.05b$ du -sh temp/
1.6G    temp

There are no directories inside data or bin_store. Basically data contains BIG text files and bin_store contains LOTS of small files containing serialized java objects.

reiserfs

Code: Select all

$time cp -R bin_store/ temp/
real    5m23.294s
user    0m0.410s
sys     0m8.805s

bash-2.05b$ time tar cf temp.tar temp

real    1m29.099s
user    0m0.545s
sys     0m6.365s

bash-2.05b$ time rm -rf temp

real    0m6.240s
user    0m0.030s
sys     0m2.400s

bash-2.05b$ time cp -R data/ temp/

real    2m11.597s
user    0m0.245s
sys     0m11.095s

ext3

Code: Select all

bash-2.05b$time cp -R bin_store/ temp/
real    3m17.108s
user    0m0.565s
sys     1m36.570s

bash-2.05b$ time tar cf temp.tar temp
real    1m37.202s
user    0m0.605s
sys     0m6.100s

bash-2.05b$ time rm -rf temp

real    0m13.892s
user    0m0.015s
sys     0m1.320s

bash-2.05b$ time cp -R data/ temp/

real    3m16.034s
user    0m0.205s
sys     0m10.965s

Not too aparent that reiserFS is a lot better. If you see any obvious flaw in this test that makes the test biased let me know.

[pilla]

Dup of the filesystem choice thread