View previous topic :: View next topic |
Author |
Message |
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Wed Mar 21, 2007 10:59 am Post subject: Best filesystem for portage? [A Study] |
|
|
I recently added a new hard drive and was doing the usual web searches to try to understand what is the best filesystems and options to use. Rathar than just read about it, I figured to make some measurements and collect some data. This post reports on the experience.
In particular I wondered how well xfs would work for portage. This was sparked by positive experiences using xfs for my music collection. But previously a quick test had indicated that portage on xfs with its default settings takes up quite a bit more space than on reiserfs-3.6, which I'm currently using. I wondered whether a different options might improve this. I was also curious how different inode sizes might affect the outcome.
Setup:
Machine:
Cpu: Amd 4400 dualcore 2.2GHz
RAM: 2 GB
Disk: Seagate ST3750640AS
OS: gentoo amd64 stable (mostly); 2.6.19-gentoo-r5 with CFQ scheduler, low-latency, 1KHz timer options
Disk speed: (hdparm -Tt /dev/sda7)
Timing cached reads: 2168 MB in 2.00 seconds = 1084.39 MB/sec
Timing buffered disk reads: 222 MB in 3.01 seconds = 73.80 MB/sec
Test preparation:
- Create a tar file of my current portage + distfiles on a different (alas somewhat slower) disk. (Note that my portage includes quite a few distfiles; a listing of the actual contents appears in a subsequent post.)
- The machine was very lightly loaded during these tests (just writing this post using kate).
The test consists of the following:
- make the filesystem with the specified options
- mount the newly-created filesystem and time untar the portage tarfile into it
- note what 'df' reports
- unmount the filesystem (to clear the kernel's cache)
- remount the filesystem and time du -s
- un- and re-mount the filesystem and time 'emerge --search notfound'
The measured data is summarized in the following tables. Subsequent posts contains the test script used and teh details of what's in the portage.tar.z I used.
I report all 3 significant digits that the time command gives. But looking at the variation in user times (which shouldn't be affected much by filesystem) it suggests that differences less than 0.1 seconds are not statistically significant.
Discussion:
On df, the blocks reported used did not always correlate with the % used. I noticed that ext3 starts with a somewhat smaller amount of total blocks than the others. Also ext3 by default reserves 5% of the blocks so the percentages reported are that much higher.
The size reported by du doesn't always correlate with df. If you order the filesystems by df(used) and compare that with the ordering from du they are totally different. I'm not sure why.
The time taken to untar the file seemed especially fast in reiser-fs, rathar fast in ext3, and particularly slow in xfs. Since the source tar-file was made on a reiser-fs I wondered whether files got placed in a particularly convenient way. So I created a new tarfile using ext3 and timed reiser-fs using that. Those numbers are given in brackets. Reiser-fs is still quite fast.
Among the xfs filesystems, the smallest inode and blocksize combination seems to be the best both in terms of size as well as speed. The speed of the small blocksizes is somewhat surprising given the rathar large amount of (biggish) distfiles that are present. I guess that is because the smaller disk footprint requires less I/O, and I/O is still the limiting factor (the smaller blocksizes do require more 'sys' time).
ext3 seems to take an disproportionally long time for a 'du'. Likewise for reiserfs when extracted from the ext3-created tar file. I wonder why that is.
In terms of emerge --search, reiser-fs is the winner with the small-blocksize xfs a close second, and the 4k blocksize ext3 not far behind that. Fragmentation is a issue at least with reiser-fs, since the very same search on my currently live filesystem takes 14.9 seconds, significantly more than the fresh-filesystem 6.047. I don't know how much of an issue it'll be with the other filesystems since I don't have a live filesystem that was banged on a lot to test against.
Summary:
Of the filesystems tested, reiser-fs 3.6 seems to be a good choice for portage. It has the smallest disk footprint and the fastest search time. Search time seems to be dependent upon file creation order as evidenced by the 10+ second search when a fresh portage was extracted from a tarfile created from ext3 compared to a 6-second search when extracted from a tarfile created on another reiser-fs. This suggests there might be variability in performance as the tree gets updated.
Xfs with a small (512-byte) blocksize seems to be another good choice. I would have recommended it without reservation except the rathar long tar extraction time is a concern. I'm not sure whether that is a artifact of the fileorder in the tarfile, or whether xfs is simply slower on writes. More study here is required.
Ext3 is the worst of the three. Good search times can be had with the large (4K) blocksize, but the disk footprint suffers significantly relative to reiser-fs and the small-blocksize xfs. There does not seem to be a tuning that gives both good speed and good disk footprint.
Conclusions:
Reiser-fs is good. Xfs with small block sizes looks very interesting and potentially good, more study is needed on why extracting the tarfile takes significantly longer. I'll be trying a xfs 512-byte blocksize portage for a while and see how that goes, how it stands up to emerge --sync, etc.
What is especially interesting is how both in xfs and in ext3 the smaller blocksizes save quite a bit of diskspace even though my portage has a rathar large amount of distfiles.
What started out to be a rathar simple "let's see" project ended up using up two whole evenings. Each experiment led to more questions, more ideas of things to measure, and a desire to include even more filesystems - I had to stop or risk never finishing! Hopefully this will be useful or interesting to others as well. |
|
Back to top |
|
|
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Wed Mar 21, 2007 11:02 am Post subject: |
|
|
Measured data:
Code: | df-reported total 1k blocks:
xfs (all) 5226918
reiserfs 5236956
ext3 (4096) 5154852
ext3 (2048) 5154508
ext3 (1024) 5153661 |
Below, the best 3 are ranked as (a), (b), (c). Multiple instances of a letter can appear when differences are not significant.
Code: | === df (Space Used) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 3910312 (75%) 3794848 (73%) 3736008 (72%) 3719204 (72%) 3346076 (64%) 3798984 (78%)
| [3345980] (a)
2048 | 3617276 (70%) 3561152 (69%) 3542428 (68%) 3523566 (73%)
|
1024 | 3451725 (67%) 3415447 (66%) 3392437 (70%) (c)
|
512 | 3385928 (65%) (b)
=== du -s (kbytes) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 3555220 3555604 3556268 3566076 3557817 3657564
| [3557817]
2048 | 3390830 3391494 3401298 3446632
|
1024 | 3318821 (b) 3328623 (c) 3355954
|
512 | 3300187 (a)
=== time tar xf (real) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 449.961 454.099 452.449 468.235 90.197 (a) 99.997 (b)
| [99.088]
2048 | 384.927 405.291 404.914 128.627 (c)
|
1024 | 362.914 367.336 209.608
|
512 | 345.779
=== time tar xf (sys) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 24.551 23.328 (c) 22.499 (b) 21.868 (a) 25.317 26.146
| [25.109]
2048 | 25.787 25.784 25.234 34.552
|
1024 | 28.371 27.990 49.994
|
512 | 32.206
=== time du -s (real) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 10.985 7.861 5.646 (b) 6.112 18.005 50.019
| [40.924]
2048 | 7.507 5.540 (a) 5.968 41.454
|
1024 | 5.772 5.835 49.368
|
512 | 5.569 (a)
=== time du -s (sys) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 4.297 2.962 2.072 (a) 2.012 (a) 2.838 2.368
| [2.832]
2048 | 3.197 2.271 (b) 2.232 (b) 2.489
|
1024 | 2.276 (b) 2.248 (b) 2.827
|
512 | 2.295 (b)
=== time emerge --search (real) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 16.628 8.937 7.111 6.569 6.047 (a) 6.692
| [10.669]
2048 | 8.926 7.222 6.774 8.504
|
1024 | 7.400 6.321 (c) 13.769
|
512 | 6.147 (b)
=== time emerge --search (sys) (seconds) ===
block | |===================== xfs inode size =====================|
size | 2048 1024 512 256 reiserfs ext3
--------+-------------------------------------------------------------------------------------------
4096 | 0.832 0.777 0.530 0.387 (a) 0.456 0.318
| [0.475]
2048 | 0.800 0.549 0.405 (b) 0.448
|
1024 | 0.563 0.443 0.668
|
512 | 0.400 (b)
|
Last edited by Akkara on Wed Mar 21, 2007 11:16 am; edited 1 time in total |
|
Back to top |
|
|
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Wed Mar 21, 2007 11:08 am Post subject: |
|
|
The test script:
Code: | DEV="/dev/==a-free-partition-here=="
umount /usr/portage
set -x
for fsys in
"mkfs.xfs -f -b size=4096 -i size=2048 -d agcount=4" \
"mkfs.xfs -f -b size=4096 -i size=1024 -d agcount=4" \
"mkfs.xfs -f -b size=4096 -i size=512 -d agcount=4" \
"mkfs.xfs -f -b size=4096 -i size=256 -d agcount=4" \
"mkfs.xfs -f -b size=2048 -i size=1024 -d agcount=4" \
"mkfs.xfs -f -b size=2048 -i size=512 -d agcount=4" \
"mkfs.xfs -f -b size=2048 -i size=256 -d agcount=4" \
"mkfs.xfs -f -b size=1024 -i size=512 -d agcount=4" \
"mkfs.xfs -f -b size=1024 -i size=256 -d agcount=4" \
"mkfs.xfs -f -b size=512 -i size=256 -d agcount=4" \
"mkfs.reiserfs" \
"mkfs.ext3 -b 4096" \
"mkfs.ext3 -b 2048" \
"mkfs.ext3 -b 1024" \
; do
echo "= = = = = = = = = = $fsys = = = = = = = = = ="
$fsys $DEV
mount $DEV /mnt/portage
time tar xfpz /s/portage2.tar.z -C /mnt/portage
df /mnt/portage
umount /mnt/portage
mount $DEV /mnt/portage
time du -s /mnt/portage
umount /mnt/portage
mount $DEV /usr/portage
time emerge --search notfound
umount /usr/portage
done
mount /usr/portage |
|
|
Back to top |
|
|
Unlucky_Alf Tux's lil' helper
Joined: 02 Feb 2003 Posts: 92
|
Posted: Fri Mar 30, 2007 8:30 am Post subject: |
|
|
Thanks for that info. I was just about to format a partition on my testserver for /usr/portage. After reading your post and some more info about reiser and xfs I decided to try xfs, because fragmentation seems to be a problem with reiser. |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Fri Mar 30, 2007 6:08 pm Post subject: |
|
|
A while ago I posted this comment about using ext2 with a small blocksize for /usr/portage. Currently "du -s /usr/portage" takes 2.6 seconds on my system (right after a remount). The footprint reported by df is 231M. |
|
Back to top |
|
|
Unlucky_Alf Tux's lil' helper
Joined: 02 Feb 2003 Posts: 92
|
Posted: Sat Mar 31, 2007 7:03 am Post subject: |
|
|
My testserver, xfs, blocksize 512k:
Code: | hdparm -tT /dev/hda
Timing cached reads: 412 MB in 2.00 seconds = 205.72 MB/sec
Timing buffered disk reads: 118 MB in 3.04 seconds = 38.77 MB/sec
|
Code: | time du -sh /usr/portage
189M /usr/portage
real 0m0.952s
user 0m0.174s
sys 0m0.778s
|
It has been synced only twice now, so there should be no fragmentation.
Edit: I just remembered, that there are some rsync_excludes so these results are not comparable. It looks like I haven't deleted those directories, so the files are there, but synced about 2-3 years ago |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|