bcachefs

Message

mkoniarz · Post by **mkoniarz** » Sat Jun 02, 2018 11:17 am

I'm curious if anyone was testing

https://bcachefs.org/

??

Would anyone be nice enough to prepare ebuild ? (bcachefs-tools)
Or where to start. maybe I could done by myself and share here?

Naib · Post by **Naib** » Sat Jun 02, 2018 1:51 pm

I am looking forward to this.
zfs is out of kernel (been burnt too many times... if my system cannot boot with a stock kernel I am not interested). btrfs seems to have messed up and redhat have their userland ontop of xfs pipedream....

I have just created a 1st pass ebuild

Code: Select all

# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=5

EGIT_NONSHALLOW=true

inherit git-r3 toolchain-funcs udev

DESCRIPTION="Tools for bcachefs"
HOMEPAGE="http://bcachefs.evilpiepirate.org/"
SRC_URI=""
EGIT_REPO_URI="https://evilpiepirate.org/git/bcachefs-tools.git"

SLOT="0"
LICENSE="GPL-2"
KEYWORDS="~amd64"
IUSE=""

RDEPEND=""
DEPEND="${RDEPEND}
	sys-apps/attr
	sys-apps/util-linux
	app-crypt/libscrypt
	dev-libs/libsodium
	sys-apps/keyutils
	dev-libs/userspace-rcu
	dev-util/pkgconfig
	sys-libs/zlib
	app-arch/zstd
"

--edit--
bug submitted https://bugs.gentoo.org/657120

msst · Post by **msst** » Sat Jun 02, 2018 9:37 pm

btrfs seems to have messed up

Maybe you have other info than I do, but it works stable and without problems since quite a while now. It is also in the kernel and not going to vanish there. Some distrios have even made it their default filesystem. RedHat decided in the opposite direction, but that won't make it vanish.

Naib · Post by **Naib** » Sat Jun 02, 2018 9:57 pm

msst wrote:
btrfs seems to have messed up
Maybe you have other info than I do, but it works stable and without problems since quite a while now. It is also in the kernel and not going to vanish there. Some distrios have even made it their default filesystem. RedHat decided in the opposite direction, but that won't make it vanish.

https://www.patreon.com/bcachefs

btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too quickly without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors who years ago hoped to one day roll out btrfs are now quietly migrating to xfs instead).

sure this is the author of bcachefs so it probably has some bias, but outright slander wouldn't last long

Post by **bunder** » Sun Jun 03, 2018 4:17 am

There was this: https://www.theregister.co.uk/2017/11/2 ... roys_data/

Also, isn't stratis just lvm/dm+xfs?

Naib · Post by **Naib** » Sun Jun 03, 2018 8:25 am

bunder wrote:There was this: https://www.theregister.co.uk/2017/11/2 ... roys_data/

Also, isn't stratis just lvm/dm+xfs?

that's bcache not bcache.

msst · Post by **msst** » Sun Jun 03, 2018 9:04 am

sure this is the author of bcachefs so it probably has some bias, but outright slander wouldn't last long

It is clearly an opinion, he seems to dislike btrfs coding design (cannot coment on this) and what is correct IMHO is that the early releases were quite alpha and produced problems.They were not tagged as stable, but people tried them and expected something else they got. And yes, development was slow as is true for many open source projects.

Since more than 3 years I am now using btrfs and it has not yet produced any problem including raid 0/10. Facebook seems to be using it in production as well. One can consider it stable with the exception of raid 5/6 (which may never get stable as these raid levels are becoming obsolete quickly).

Also, isn't stratis just lvm/dm+xfs?

Red Hats decision seems somewhat political and I have no idea what stratis will be. If they make it just a repackaging of lvm+dm+xfs then it won't be able to compete. Who knows what the future will bring, but brtrfs will not go away from the kernel for many years, so that is no reason to avoid it.
And in the moment btrfs is much more established, adopted and tried than bcachefs. Kent will have to do some catch up work. Making new filesystems is certainly a daunting job.

Post by **bunder** » Sun Jun 03, 2018 9:25 am

msst wrote:Who knows what the future will bring, but brtrfs will not go away from the kernel for many years, so that is no reason to avoid it.

I'm quite happy with openzfs and I don't care whether it hits mainline or not. I also don't care if Oracle eventually ports their v28-v43 closed source zfs to their "unbreakable" redhat clone.

Call me biased but I'm quite happy with my storage situation right now.

But good for them, there is always more room for competition.

Naib · Post by **Naib** » Sun Jun 03, 2018 10:48 am

msst wrote:
sure this is the author of bcachefs so it probably has some bias, but outright slander wouldn't last long
It is clearly an opinion, he seems to dislike btrfs coding design (cannot coment on this) and what is correct IMHO is that the early releases were quite alpha and produced problems.They were not tagged as stable, but people tried them and expected something else they got. And yes, development was slow as is true for many open source projects.

Since more than 3 years I am now using btrfs and it has not yet produced any problem including raid 0/10. Facebook seems to be using it in production as well. One can consider it stable with the exception of raid 5/6 (which may never get stable as these raid levels are becoming obsolete quickly).

I agree his statements are opinionated, especially as he has a "competitor" ... but this isn't the 1st time I have come across statement with respect to poor direction with btrfs. RH dropping it was more than likely politically driven BUT equally something had to aggravate them, ie BTRFS not producing what was needed...

Also, isn't stratis just lvm/dm+xfs?
Red Hats decision seems somewhat political and I have no idea what stratis will be. If they make it just a repackaging of lvm+dm+xfs then it won't be able to compete. Who knows what the future will bring, but brtrfs will not go away from the kernel for many years, so that is no reason to avoid it.
And in the moment btrfs is much more established, adopted and tried than bcachefs. Kent will have to do some catch up work. Making new filesystems is certainly a daunting job.

My concerns with Stratis is it will hook deeply into systemd as the aim appears to use a proven filesystem and bolt onto it "next gen" features in userland via daemons...

So essentially the next set of filesystems are...zfs,btrfs,stratis,bcachefs
ZFS would be the preferred option but the license issue stops it being mainlined (there is talks though...). For some this isn't an issue. HOWEVER I have been burnt in the past (ReiserFS on Mandrake, FakeRAID on Gentoo) and as such anything needed to boot my system must be in the vanilla source.
BTRFS looks like a possible alternative to ZFS but I keep coming across concerns ( straight from there wiki... WARNING: kernels 4.14.25 - 4.14.27 and 4.15.8 - 4.15.9 on big endian machines will damage a filesystem after first umount. https://btrfs.wiki.kernel.org/index.php/Main_Page) Its thinks like that that concerns me "don't use those kernels" ... and when the next issue comes along?

ZFS and BTRFS being developed by Oracle is also oO seems a waste of resource (I know btrfs was started by oracle and they then acquired zfs via sun but still...)

STRATIS... political and I have fears of hooks into systemd. Whether it is an open threat to get BTRFS or ZFS-GPL who knows...

BCACHEFS its intriguing I do like how they have targeted stability and core FS before jumping onto shinies. That was always my concern with BTRFS... they kept pushing the latest buzword stuff but then breaking diskformat or breaking the format

Post by **bunder** » Sun Jun 03, 2018 11:47 am

Not to derail the thread, but openzfs is actually maintained quite well in gentoo. I talk to several gentoo developers (+ foundation members) and users on a daily basis through freenode #zfsonlinux.

Also, openzfs hasn't been maintained by Sun/Oracle since they closed their source in 2005, it's maintained by former Sun employees, illumos (formerly known as OpenSolaris) developers, freebsd developers, linux developers, mac developers, and most importantly, the US Government. (Not to mention countless randoms with no programming experience.)

(I guess technically speaking, Oracle never maintained it, one of the first things they did was to close it, so we shouldn't even be using the O word.

)

That said, if you ever do play around with it and have questions, you know where to find me.

happosai · Post by **happosai** » Mon Jun 11, 2018 8:54 pm

msst wrote: Since more than 3 years I am now using btrfs and it has not yet produced any problem including raid 0/10. Facebook seems to be using it in production as well. One can consider it stable with the exception of raid 5/6 (which may never get stable as these raid levels are becoming obsolete quickly).

Also, isn't stratis just lvm/dm+xfs?
Red Hats decision seems somewhat political and I have no idea what stratis will be. If they make it just a repackaging of lvm+dm+xfs then it won't be able to compete. Who knows what the future will bring, but brtrfs will not go away from the kernel for many years, so that is no reason to avoid it.
And in the moment btrfs is much more established, adopted and tried than bcachefs. Kent will have to do some catch up work. Making new filesystems is certainly a daunting job.

Btrfs has been in development for over ten years now, and while the basic stuff is being considered stable, many of the more advanced features are in development, not really finalized yet and some are even unstable - check this out: https://btrfs.wiki.kernel.org/index.php/Status. If you want an even more sceptic analysis of its status, look here: https://github.com/mosteo/btrfs-status

The show stopper for enterprise use is in many cases that Btrfs doesn't support triple parity, while ZFS has been offering this feature since ages. Another show stopper for using btrfs is that it just plainly sucks with VM images and databases, in fact the official recommendation for both is to switch COW off then, because otherwise your file might fragment in literelly ten thousand of pieces or more. But what's the point in using VM images/database storage on a COW file system, when you cannot use the COW features then? For both usage types you are way better of with XFS, or ext4 - in that order.

Also another problem with the still unstable state of btrfs is that putting it on a server means that you should be able to roll out the latest and greatest kernel version every release cycle, because it may contain important bug fixes and performance optimizations. The thing is if your preferred Linux distribution doesn't do backports, than choosing Btrfs might not be such an exactly brillant idea, because when you are running into issues the developers will tell you to switch to the latest and greatest kernel.

Facebook by the way has been testing it in production, but mostly for the deployment and management of system partitions, and not much more. Btrfs on Red Hat has never been a first class citizen, but yes, they lost faith in it completely.

If you want to get real insights what using btrfs in production use meaned, just look at CoreOS. They rolled it out as their default file system in 2014, and the complaints and problems about it skyrocketed to such a boiling point, that they ditched it entirely back then and switched to ext4 + overlayfs early in 2015. The most typical complaints were out of disk space errors, metadata rebalancing problems, and generally slow performance compared to other file systems (https://lwn.net/Articles/627232/). Rebalanciing problems are even an issue in 2018: https://www.spinics.net/lists/linux-btrfs/msg74892.html

Russel Coker from Australia is writing in his personal blog about his experience with Btrfs on his own machines, which he started bloggin about in 2012 and you might also want to check here: https://etbe.coker.com.au/ In the first few years he had a major hiccup almost every six months or so, were he had to ask the mailing list for help, but then it became more reliable and it slowed down.

I also used it several times on my own, and it always ran okay for some weeks then, when problems started to occur and the fun stuff happened. And RAID1 is now nothing fancy in particular. I always had a backup of my data, just in case, and I always needed it.

Having said that, Btrfs is fine to play around with in my opinion if you are using it with throwaway data only, or you get a regular backup of your data. I would never, ever use it on a production level system.

Synology though does use it since over two years as standard file systems, and they seem do be quite happy with it.

To make it short: if you really need a COW file system on Linux, then take ZFS. If you have issues with it on a Linux machine due to CCDL vs GPL, then switch over to FreeBSD. Btrfs has been a nice idea, but is still to much construction area, and a convoluted mess in reality which I would not trust my data with. And BcacheFS might be what could become the new king, but give it at least two or three years, because they are still lacking some basic features like scrubbing. The thing though is that many people, just like myself, have lost their faith in Btreefs completely, so time is not such a pressing issue for BcacheFS at the moment as some might think.

And by the way Oracle recently in February this year completely unexpected relicensed another cron jewel from Sun, dtrace, under the GPL. Maybe ZFS might follow, who knows?

steveL · Post by **steveL** » Tue Jun 12, 2018 12:41 pm

I like the sound of it; I've always wondered why people don't talk about the bcache more, since it underpins everything, and has been around since the beginning of UNIX. (cf: Lions, Bach.)

It's the original "page cache", afaic (and I don't think you really need another one in-kernel; just memory areas or "extents".)

Ant P. · Post by **Ant P.** » Tue Jun 12, 2018 6:32 pm

Bah. Still waiting for Tux3.

Naib · Post by **Naib** » Sun Dec 02, 2018 12:29 pm

Update:
post to LKML. https://lkml.org/lkml/2018/12/2/46 possible push for mainlining
Patreon update https://www.patreon.com/posts/status-update-23029978

Naib · Post by **Naib** » Mon Apr 22, 2019 12:11 pm

update:

https://www.patreon.com/bcachefs

Apr 20 at 4:39am
Fully persistent allocation info is finally done

Finally! It was a huge effort, but it's done and pushed out.

This means that when mounting a filesystem - even after an unclean shutdown - we don't have to walk all the metadata anymore, because it's always updated in a transactional manner and kept fully consistent in the b-tree.

There may be a performance regression for now on multithreaded write workloads, due to lock contention on the alloc btree. But, that will go away when I implement the new btree key cache code (it'll generalize the current deferred btree updates code, used for inode updates).

If anyone does notice a performance regression, post about it so I'll know to prioritize that.

And, this is the last checkbox - besides catching up on the bug reports - on my list before pushing bcachefs upstream. So, expect to see movement on that front over the coming months.

getting there

Website · Post by **Zucca** » Mon Apr 22, 2019 4:18 pm

Is bcachefs as painless as btrfs with multi-device storage pools?
I mean with btrfs I can simply add drives to the storage pool, and btrfs spreads the data so that I lose storage space as little as possible while also maintaining redundancy.

depontius · Post by **depontius** » Tue Apr 23, 2019 12:33 pm

Zucca wrote:Is bcachefs as painless as btrfs with multi-device storage pools?

Another way of putting it, does bcachefs engage in layer-breaking like btrfs. The way one would normally do the multi-device stuff is with lvm2 at a layer below the filesystem. With btrfs they break that layering model, taking advantage of that break. So you're really asking if bcachefs sits on lvm2, without layer-breaking, or if it does layer-breaking as btrfs does. I'm not meaning to condemn with the term "layer-breaking", just describe.

Post by **pjp** » Tue Apr 23, 2019 10:20 pm

depontius wrote:
Zucca wrote:Is bcachefs as painless as btrfs with multi-device storage pools?
Another way of putting it, does bcachefs engage in layer-breaking like btrfs. The way one would normally do the multi-device stuff is with lvm2 at a layer below the filesystem. With btrfs they break that layering model, taking advantage of that break. So you're really asking if bcachefs sits on lvm2, without layer-breaking, or if it does layer-breaking as btrfs does. I'm not meaning to condemn with the term "layer-breaking", just describe.

I think what he is asking is how well bcachefs handles growing or shrinking a storage pool. The issue of layer-breaking isn't really relevant unless someone has an arbitrary requirement to use LVM.

That said, the website mentions that it handles multiple devices, but does not explain how it will handle modifying storage pools. I'm pretty sure it mentioned handling drive replacement. So it would seem that like btrfs and ZFS, bcacehfs does not rely on LVM. A different question might be whether or not bcachefs also supports "legacy solutions*" such as LVM. (* After having used ZFS, I can't think of a more accurate way to think of LVM... I'm not aware of any efforts to improve it's capabilities to close the gap with btrfs/ZFS/bcachefs?).

It is important to keep in mind that this is not ready for use. From what I've read, it seems like the intent is for bcachefs to be able to grow and shrink a storage pool. I may have read that in one of the emails about it, or I may be just hopeful. But I seem to recall thinking that it was going to be "better" than btrfs and also avoid ZFS' storage pool modification shortcomings.

Naib · Post by **Naib** » Tue Apr 23, 2019 11:22 pm

totally, this is not production ready but for all intent and purposes it appears stable. It appears to be built on an extremely sound and stable foundation

I'm tempted to build my /var/tmp drive with this to see how it fairs... that does mean I need to fix the ebuild I knocked together for the bcachefs-tools ( https://bugs.gentoo.org/657120 )

as to adding/removing drives, its one of its aims, just don't know the state of it. The goals of BcacheFS seem sound and the development of it seems well thought out. When you look at the so called "next gen FS" for linux, this appears to be ticking alot of boxes after btrfs fizzled out and the relicensing of zfs is nonexistent

Website · Post by **Zucca** » Wed Apr 24, 2019 2:49 pm

Naib wrote:When you look at the so called "next gen FS" for linux, this appears to be ticking alot of boxes after btrfs fizzled out and the relicensing of zfs is nonexistent

XFS+lvm2 also ticks many boxes too. lvm2 can be very flexible and can even do raid (via underlying mdraid). But as to how simple it is to use... I have no experience.

Post by **pjp** » Wed Apr 24, 2019 4:00 pm

Naib wrote:When you look at the so called "next gen FS" for linux, this appears to be ticking alot of boxes after btrfs fizzled out and the relicensing of zfs is nonexistent

I didn't expect much from the teaser about the ZFS license change, but I wonder how stable the ZFS ecosystem would be. The versioning scheme for features sounds like a good idea for development flexibility, but also seems like it could lead to some problems. If not for having to maintain patches, I'd probably use it.

Website · Post by **Zucca** » Fri Jun 14, 2019 7:18 pm

Progress: https://www.phoronix.com/scan.php?page= ... tures-Done

Naib · Post by **Naib** » Sun Jun 16, 2019 10:24 am

yup I saw that. Linus was highly critical of what was provided and also the state it was provided. it is constructive criticism and now its a case of tidying up the code for acceptance. this will take some time BUT its happening

aidanjt · Post by **aidanjt** » Sun Jun 16, 2019 11:20 am

Zucca wrote:XFS+lvm2 also ticks many boxes too. lvm2 can be very flexible and can even do raid (via underlying mdraid). But as to how simple it is to use... I have no experience.

I wouldn't call it simple to use, but it does make managing logical volumes easier than not having it. It has caching support as well, now.

Website · Post by **Zucca** » Fri Sep 15, 2023 8:06 am

More progress: https://www.phoronix.com/news/Bcachefs-In-Linux-Next