Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Best filesystem for portage? [A Study]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Wed Mar 21, 2007 10:59 am    Post subject: Best filesystem for portage? [A Study] Reply with quote

I recently added a new hard drive and was doing the usual web searches to try to understand what is the best filesystems and options to use. Rathar than just read about it, I figured to make some measurements and collect some data. This post reports on the experience.

In particular I wondered how well xfs would work for portage. This was sparked by positive experiences using xfs for my music collection. But previously a quick test had indicated that portage on xfs with its default settings takes up quite a bit more space than on reiserfs-3.6, which I'm currently using. I wondered whether a different options might improve this. I was also curious how different inode sizes might affect the outcome.

Setup:

Machine:
Cpu: Amd 4400 dualcore 2.2GHz
RAM: 2 GB
Disk: Seagate ST3750640AS
OS: gentoo amd64 stable (mostly); 2.6.19-gentoo-r5 with CFQ scheduler, low-latency, 1KHz timer options

Disk speed: (hdparm -Tt /dev/sda7)
Timing cached reads: 2168 MB in 2.00 seconds = 1084.39 MB/sec
Timing buffered disk reads: 222 MB in 3.01 seconds = 73.80 MB/sec


Test preparation:
- Create a tar file of my current portage + distfiles on a different (alas somewhat slower) disk. (Note that my portage includes quite a few distfiles; a listing of the actual contents appears in a subsequent post.)
- The machine was very lightly loaded during these tests (just writing this post using kate).

The test consists of the following:
- make the filesystem with the specified options
- mount the newly-created filesystem and time untar the portage tarfile into it
- note what 'df' reports
- unmount the filesystem (to clear the kernel's cache)
- remount the filesystem and time du -s
- un- and re-mount the filesystem and time 'emerge --search notfound'

The measured data is summarized in the following tables. Subsequent posts contains the test script used and teh details of what's in the portage.tar.z I used.

I report all 3 significant digits that the time command gives. But looking at the variation in user times (which shouldn't be affected much by filesystem) it suggests that differences less than 0.1 seconds are not statistically significant.

Discussion:

On df, the blocks reported used did not always correlate with the % used. I noticed that ext3 starts with a somewhat smaller amount of total blocks than the others. Also ext3 by default reserves 5% of the blocks so the percentages reported are that much higher.

The size reported by du doesn't always correlate with df. If you order the filesystems by df(used) and compare that with the ordering from du they are totally different. I'm not sure why.

The time taken to untar the file seemed especially fast in reiser-fs, rathar fast in ext3, and particularly slow in xfs. Since the source tar-file was made on a reiser-fs I wondered whether files got placed in a particularly convenient way. So I created a new tarfile using ext3 and timed reiser-fs using that. Those numbers are given in brackets. Reiser-fs is still quite fast.

Among the xfs filesystems, the smallest inode and blocksize combination seems to be the best both in terms of size as well as speed. The speed of the small blocksizes is somewhat surprising given the rathar large amount of (biggish) distfiles that are present. I guess that is because the smaller disk footprint requires less I/O, and I/O is still the limiting factor (the smaller blocksizes do require more 'sys' time).

ext3 seems to take an disproportionally long time for a 'du'. Likewise for reiserfs when extracted from the ext3-created tar file. I wonder why that is.

In terms of emerge --search, reiser-fs is the winner with the small-blocksize xfs a close second, and the 4k blocksize ext3 not far behind that. Fragmentation is a issue at least with reiser-fs, since the very same search on my currently live filesystem takes 14.9 seconds, significantly more than the fresh-filesystem 6.047. I don't know how much of an issue it'll be with the other filesystems since I don't have a live filesystem that was banged on a lot to test against.

Summary:

Of the filesystems tested, reiser-fs 3.6 seems to be a good choice for portage. It has the smallest disk footprint and the fastest search time. Search time seems to be dependent upon file creation order as evidenced by the 10+ second search when a fresh portage was extracted from a tarfile created from ext3 compared to a 6-second search when extracted from a tarfile created on another reiser-fs. This suggests there might be variability in performance as the tree gets updated.

Xfs with a small (512-byte) blocksize seems to be another good choice. I would have recommended it without reservation except the rathar long tar extraction time is a concern. I'm not sure whether that is a artifact of the fileorder in the tarfile, or whether xfs is simply slower on writes. More study here is required.

Ext3 is the worst of the three. Good search times can be had with the large (4K) blocksize, but the disk footprint suffers significantly relative to reiser-fs and the small-blocksize xfs. There does not seem to be a tuning that gives both good speed and good disk footprint.


Conclusions:

Reiser-fs is good. Xfs with small block sizes looks very interesting and potentially good, more study is needed on why extracting the tarfile takes significantly longer. I'll be trying a xfs 512-byte blocksize portage for a while and see how that goes, how it stands up to emerge --sync, etc.

What is especially interesting is how both in xfs and in ext3 the smaller blocksizes save quite a bit of diskspace even though my portage has a rathar large amount of distfiles.

What started out to be a rathar simple "let's see" project ended up using up two whole evenings. Each experiment led to more questions, more ideas of things to measure, and a desire to include even more filesystems - I had to stop or risk never finishing! Hopefully this will be useful or interesting to others as well.
Back to top
View user's profile Send private message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Wed Mar 21, 2007 11:02 am    Post subject: Reply with quote

Measured data:

Code:
df-reported total 1k blocks:
xfs (all)   5226918
reiserfs   5236956
ext3 (4096)   5154852
ext3 (2048)   5154508
ext3 (1024)   5153661



Below, the best 3 are ranked as (a), (b), (c). Multiple instances of a letter can appear when differences are not significant.


Code:
=== df (Space Used) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       3910312 (75%)   3794848 (73%)   3736008 (72%)   3719204 (72%)   3346076 (64%)   3798984 (78%)
        |                                                                       [3345980] (a)
2048    |                       3617276 (70%)   3561152 (69%)   3542428 (68%)                   3523566 (73%)
        |
1024    |                                       3451725 (67%)   3415447 (66%)                   3392437 (70%) (c)
        |
 512    |                                                       3385928 (65%) (b)


=== du -s (kbytes) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       3555220         3555604         3556268         3566076         3557817         3657564
        |                                                                       [3557817]
2048    |                       3390830         3391494         3401298                         3446632
        |
1024    |                                       3318821 (b)     3328623 (c)                     3355954
        |
 512    |                                                       3300187 (a)


=== time tar xf (real) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       449.961         454.099         452.449         468.235         90.197 (a)      99.997 (b)
        |                                                                       [99.088]
2048    |                       384.927         405.291         404.914                         128.627 (c)
        |
1024    |                                       362.914         367.336                         209.608
        |
 512    |                                                       345.779



=== time tar xf (sys) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       24.551          23.328 (c)      22.499 (b)      21.868 (a)      25.317          26.146
        |                                                                       [25.109]
2048    |                       25.787          25.784          25.234                          34.552
        |
1024    |                                       28.371          27.990                          49.994
        |
 512    |                                                       32.206



=== time du -s (real) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       10.985          7.861           5.646 (b)       6.112           18.005          50.019
        |                                                                       [40.924]
2048    |                       7.507           5.540 (a)       5.968                           41.454
        |
1024    |                                       5.772           5.835                           49.368
        |
 512    |                                                       5.569 (a)


=== time du -s (sys) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       4.297           2.962           2.072 (a)       2.012 (a)       2.838           2.368
        |                                                                       [2.832]
2048    |                       3.197           2.271 (b)       2.232 (b)                       2.489
        |
1024    |                                       2.276 (b)       2.248 (b)                       2.827
        |
 512    |                                                       2.295 (b)


=== time emerge --search (real) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       16.628          8.937           7.111           6.569           6.047 (a)       6.692
        |                                                                       [10.669]
2048    |                       8.926           7.222           6.774                           8.504
        |
1024    |                                       7.400           6.321 (c)                       13.769
        |
 512    |                                                       6.147 (b)



=== time emerge --search (sys) (seconds) ===

block   |       |===================== xfs inode size =====================|
size    |       2048            1024            512             256             reiserfs        ext3
--------+-------------------------------------------------------------------------------------------
4096    |       0.832           0.777           0.530           0.387 (a)       0.456           0.318
        |                                                                       [0.475]
2048    |                       0.800           0.549           0.405 (b)                       0.448
        |
1024    |                                       0.563           0.443                           0.668
        |
 512    |                                                       0.400 (b)



Last edited by Akkara on Wed Mar 21, 2007 11:16 am; edited 1 time in total
Back to top
View user's profile Send private message
Akkara
Bodhisattva
Bodhisattva


Joined: 28 Mar 2006
Posts: 6702
Location: &akkara

PostPosted: Wed Mar 21, 2007 11:08 am    Post subject: Reply with quote

The test script:
Code:
DEV="/dev/==a-free-partition-here=="
umount /usr/portage
set -x
for fsys in
        "mkfs.xfs -f -b size=4096 -i size=2048 -d agcount=4"    \
        "mkfs.xfs -f -b size=4096 -i size=1024 -d agcount=4"    \
        "mkfs.xfs -f -b size=4096 -i size=512 -d agcount=4"     \
        "mkfs.xfs -f -b size=4096 -i size=256 -d agcount=4"     \
        "mkfs.xfs -f -b size=2048 -i size=1024 -d agcount=4"    \
        "mkfs.xfs -f -b size=2048 -i size=512 -d agcount=4"     \
        "mkfs.xfs -f -b size=2048 -i size=256 -d agcount=4"     \
        "mkfs.xfs -f -b size=1024 -i size=512 -d agcount=4"     \
        "mkfs.xfs -f -b size=1024 -i size=256 -d agcount=4"     \
        "mkfs.xfs -f -b size=512 -i size=256 -d agcount=4"      \
        "mkfs.reiserfs"        \
        "mkfs.ext3 -b 4096"     \
        "mkfs.ext3 -b 2048"     \
        "mkfs.ext3 -b 1024"     \
    ; do
        echo "= = = = = = = = = = $fsys = = = = = = = = = ="
        $fsys $DEV
        mount $DEV /mnt/portage
        time tar xfpz /s/portage2.tar.z -C /mnt/portage
        df /mnt/portage
        umount /mnt/portage
        mount $DEV /mnt/portage
        time du -s /mnt/portage
        umount /mnt/portage
        mount $DEV /usr/portage
        time emerge --search notfound
        umount /usr/portage
done
mount /usr/portage
Back to top
View user's profile Send private message
Unlucky_Alf
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2003
Posts: 92

PostPosted: Fri Mar 30, 2007 8:30 am    Post subject: Reply with quote

Thanks for that info. I was just about to format a partition on my testserver for /usr/portage. After reading your post and some more info about reiser and xfs I decided to try xfs, because fragmentation seems to be a problem with reiser.
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2508
Location: Silver City, NM

PostPosted: Fri Mar 30, 2007 6:08 pm    Post subject: Reply with quote

A while ago I posted this comment about using ext2 with a small blocksize for /usr/portage. Currently "du -s /usr/portage" takes 2.6 seconds on my system (right after a remount). The footprint reported by df is 231M.
Back to top
View user's profile Send private message
Unlucky_Alf
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2003
Posts: 92

PostPosted: Sat Mar 31, 2007 7:03 am    Post subject: Reply with quote

My testserver, xfs, blocksize 512k:
Code:
hdparm -tT /dev/hda
 Timing cached reads:   412 MB in  2.00 seconds = 205.72 MB/sec
 Timing buffered disk reads:  118 MB in  3.04 seconds =  38.77 MB/sec

Code:
time du -sh /usr/portage
189M    /usr/portage
real    0m0.952s
user    0m0.174s
sys     0m0.778s

It has been synced only twice now, so there should be no fragmentation.

Edit: I just remembered, that there are some rsync_excludes so these results are not comparable. It looks like I haven't deleted those directories, so the files are there, but synced about 2-3 years ago 8)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum