New filesystem: Btrfs!

[Dont Panic]

@BlueFusion & @dufeu:

Both of you guys seem like you're really strapped for space with files that won't see much benefit from compression.

You need to keep in mind that Btrfs currently has a 90% disk space "circuit-breaker" which caps disk occupancy at 90% of total disk space. Your storage device will report no space remaining when it hits the 90% mark.

This "circuit-breaker" has probably outlived it's usefulness, but it's intention was to allow a buffer of space while the kinks in the ENOSPC (enough space, or disk full) code were worked out.

I believe the Btrfs developers feel like this 90% cap should not be an issue for people using Btrfs at this stage of development, since people are primarily using it for testing and evaluation.

Edited February 4, 2010 to Correct/Retract/Annotate:
Correction: There is no 90% disk space "circuit-breaker". I was confused by the amount of space reserved for metadata, which is not reported by the 'df' command, and was at least 10% in my trials.

[dufeu]

[BlueFusion]

Hmm, well I hope they get rid of the cap soon since ENOSPC handling is in place now.

But here's a few tests anyway. This test is on an mkv movie file that is 8.6GB (9,226,840,617 bytes). The test is using rsync to copy it from an ext4 formatted drive to a separate pool of partitions across 2 different drives.

The first transfer is using the compress option. It took 4:02 minutes to transfer. Here's the difference in physical storage used:

[Shining Arcanine]

[kernelOfTruth]

[BlueFusion]

//damn this git pull is taking forever//

Oh I understand there's plenty of kinks still in it. That's why I'm using it for stuff I feel comfortable losing (i.e. portage ebuilds and distfiles and stuff I don't access often and have backed up). But the fact that btrfs could make for an easy JBOD setup to store my 1.5-2TB of music and videos is what is most appealing to me. Right now having my separate ext4 partitions is way to annoying and have gotten quite fragmented.
_________________
Desktop: i7-940 @ 2.93GHz, ASUS P6T Deluxe, 6GB RAM, 2x 160GB, 2x 500GB, 1x 1TB Seagate Barracuda SATA, nVidia 8800GTS 512MB, Hauppage PVR-250

[Dont Panic]

[BlueFusion]

See my completed results with compress-force above.
_________________
Desktop: i7-940 @ 2.93GHz, ASUS P6T Deluxe, 6GB RAM, 2x 160GB, 2x 500GB, 1x 1TB Seagate Barracuda SATA, nVidia 8800GTS 512MB, Hauppage PVR-250

[BlueFusion]

Since I want to compare the compress and compress-force options as colsely as possible, I'm re-running both on a file of the same size with the btrfs-unstable git version.

Here's the new results with a file that is 4.4GB (4,695,212,015 bytes) in size.

This took 1:50 to rsync using the compress option.

[Dont Panic]

Right now, the 'df' results on a Btrfs storage device aren't perfect. You can expect a small amount of variation in the reported result from trial to trial.

You are probably looking at an equivalent result in space usage from your two trials.

The 'du' command seems to give me more constant results.

The COW (Copy On Write) infrastructure in Btrfs makes it difficult to get the same kind of nice, shiny, clean, reproducible 'df' results you would see on something like ext2.

[BlueFusion]

Yeah I suppose that would skew the results a bit. I believe I saw the beginnings of a patch for df to work nicely with btrfs in the mail archive. I'd have to go look.
_________________
Desktop: i7-940 @ 2.93GHz, ASUS P6T Deluxe, 6GB RAM, 2x 160GB, 2x 500GB, 1x 1TB Seagate Barracuda SATA, nVidia 8800GTS 512MB, Hauppage PVR-250

[Dont Panic]

You've got me curious now.

I saw those patches on the M/L also. Now you've captured my interest in playing with them.

I'm also looking for tools to explain the 90% maximum space utilization. There may be more to that issue than I know right now (other than it definitely exists).

[Dont Panic]

I've been playing around with filling up my Btrfs partitions to try to get a better idea of how much overhead may be involved, and I'm seeing surprising and somewhat disappointing results.

When using Btrfs in root-partition service, I'm getting 'disk-full' at between 68%-81% of disk usage as reported by the 'df' command.

I was wondering if anybody else is seeing similar performance in terms of disk full limitations. I hope I'm missing something or going about this wrong.

I usually have several bootable partitions on my computers, with different OS and/or experiments installed on each partition.

Currently, I have one Gentoo installation installed on Btrfs (with compression) on the root partition, and two Sabayon installations on two computers installed on Btrfs (one with compression, one without).

I used the dd command to fill these partitions to the 'disk full' point with random data to see where Btrfs quits.

[devsk]

[kernelOfTruth]

[Dont Panic]

I've been researching the issue with filling up my disk before I expected, and I've received some responses from the developers on the Mailing List (see Re: Btrfs Partition Reaches 'Disk Full' Early)

I'm disappointed in my findings.

It seems that the space used by metadata is not shown using the 'df' command. And the metadata can take up a considerable amount of space. And by considerable, I'm personally seeing metadata reserve between 20-30% of the disk when I've used btrfs as my root partition.

About the only way I know to see how much space the metadata is reserving (aside from filling your volume with files) is to apply some recent patches published by Josef Bacik (one for a btrfs-progs patch, and one for a kernel patch). You will have to manually apply these patches, since they are against a more experimental btrfs kernel and progs than we're using.

The default is for the metadata to be duplicated. You can prevent this by formatting your partition with '-m single', and this will cut metadata in half. But I quite frankly don't know the implications for data integrity when disabling duplication. I've asked on the M/L, but the answer didn't give me a good feel for the increased risk.

There's also a possibility they will try to tune the amount of space reserved for metadata. I found that btrfs was usually reserving about double the amount of metadata space it was actually using (assuming I'm understanding the information in the metadata info tool correctly).

But I find I'm forced to completely reconsider my impressions that btrfs is space efficient in storing data.

I will also need to clarify/retract some things I've posted earlier in this thread.

Correction #1) Earlier, I stated that there was a 90% utilization 'circuit breaker'. That is incorrect. They used to have a 'circuit breaker', but that has been removed. I was confused by the space reserved for metadata which is not reported by the 'df' command, and which was a minimum of 10% in my tests with very simple data.

Correction #2) I earlier asserted that I could find no difference in space used when formatting with single metadata (-m single). This was an incorrect assumption since the space reserved for metadata is not reported by 'df'. Indeed, you will cut the space reserved for metadata in half by using single metadata. But BE CAREFUL! I'll repeat again that I'm uncertain of the implications of going with single metadata.

[Shining Arcanine]

Regarding the single metadata, is it not the case that ext4 and other stable filesystems do not support having multiple copies of metadata in the first place?

Assuming that they only store their metadata once, I see no reliability issue in having btrfs do the same, but that is just my thinking.

[Dont Panic]

I should probably be careful with my wording. I don't want to be accused of spreading FUD about Btrfs.

With regard to metadata duplication, I've had a couple of hints as to it's purpose, but for the most part I truly don't know if it's important or not.

Supposedly, for normal problems like a power interruption, single metadata is fine. I believe it helps when you have a problem with bad bit's being written to the disk. But if you aren't already running in a RAID mode, you are probably already screwed once the situation gets to that stage.

My concerns actually stem from some of the things I'm seeing in ext4, where they are finally deciding that barriers may be important after almost of year of being 'stable'. I'm not sure if single metadata is a similar situation or not.

Anyhow, I'm not letting that stop me from running one of my root partitions with single metadata.

I've also reformatted my root partition so as to run it only with the freshest Btrfs code. As I've said before, Btrfs is still a moving target at this point, and it's possible some of my problems stemmed from using prior revisions of the Btrfs code on those partitions.

[devsk]

I am running 2.6.33-rc6, with BTRFS in a RAID0 on two drives. I was able to fill it all the way upto within 95% of the capacity reported by df and I did not see any problems. I got out of space at around 95%, which is acceptable to me (all FSs have been reserving 5% by default, so sort of live with that). Note the df difference between an ext4 and btrfs FS when you create the filesystem: btrfs consumes about 28KB while ext4 eats up 150MB, not to mention that total capacity reported by df is also lower for ext4.

Is the metadata handling of the RAID0 different from the metadata handling of single drives?

[Dont Panic]

[Letharion]

I very recently installed a new system. I read at least all posts on this side of 2010, to get a picture of the current status of btrfs. I the figured, what the heck, and went for it.

I'm posting to report that I have now happily been running btrfs on my root partition with the currently stable 31-r6 kernel for a few days. So far so good.

I'm a bit surprised, because reading posts above, it seems like I should need special options to use btrfs on /, which wasn't necessary.

[Shining Arcanine]

[Letharion]

[Dont Panic]

[Letharion]