Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
New filesystem: Btrfs! (cont.)
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3 ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Thu Jun 03, 2010 9:57 pm    Post subject: Reply with quote

Edward Shishkin had an interesting post to the Btrfs Mailing List: Unbound(?) internal fragmentation in Btrfs

If I understand this post correctly, Edward is challenging some basic elements of how B-tree sorting is implemented in Btrfs.

I'll be curious to see what follow-up posts will be made to this Mailing List thread.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jun 03, 2010 10:12 pm    Post subject: Reply with quote

Dont Panic wrote:
Edward Shishkin had an interesting post to the Btrfs Mailing List: Unbound(?) internal fragmentation in Btrfs

If I understand this post correctly, Edward is challenging some basic elements of how B-tree sorting is implemented in Btrfs.

I'll be curious to see what follow-up posts will be made to this Mailing List thread.
Ouch! I would like to see the replies as well!
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Thu Jun 03, 2010 10:44 pm    Post subject: Reply with quote

devsk wrote:
Dont Panic wrote:
Edward Shishkin had an interesting post to the Btrfs Mailing List: Unbound(?) internal fragmentation in Btrfs

If I understand this post correctly, Edward is challenging some basic elements of how B-tree sorting is implemented in Btrfs.

I'll be curious to see what follow-up posts will be made to this Mailing List thread.
Ouch! I would like to see the replies as well!


Awesome ! :lol:

I hope the devs aren't pissed too much ;)

anyways constructive criticism should always be welcome ^^

seems like he's officially working for Redhat czech now - congrats Edward ! :D
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jun 03, 2010 11:18 pm    Post subject: Reply with quote

OK, here is another gem:

http://www.spinics.net/lists/linux-btrfs/msg05108.html

The scary parts:

> a btrfsck run on a 2T volume (4 disks) on a QNAP appliance (512M ram) got
> killed by Mr. OOM Killer. Initially, I was quite surprised. I'm only
> moderately surprised now since it might well be that I forgot to enable
> swap.
>

Josef: Yes, btrfsck keeps the entire extent tree in memory, so the bigger the fs, the
more RAM it's going to use.

(devsk: what happens if the entire extent tree doesn't fit in memory? I have only 1GB of RAM e.g.)

> And 3rd observation: btrfsck apparently doesn't correct this kind of error.
> Running btrfsck again still shows the error.
>

Josef: Yeah btrfsck doesn't fix problems yet. Thats being worked on. Thanks,

why give the illusion of fixing the issues? call it btrfs-find-errors-but-screw-you-if-there-are-some instead of btrfsck because an fsck means that it can correct some FS errors.

Man! I feel terrible that I am entrusting my data to this FS. This is miles away from production. What happens if I run into issues? Who is going to help if even fsck can't fix the errors?
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Fri Jun 04, 2010 4:01 pm    Post subject: Reply with quote

Watching the Btrfs Mailing List, I almost develop the impression that the volume of fixes and patches is overwhelming the development group.

I know there are other lines of communication, but I see several patch sets floating across the Mailing List to address issues that never seem to make it into the main code tree. One area in particular is btrfs-progs, which will probably see major restructuring in the future.

I guess the idea in bringing Btrfs into the main Linux kernel tree was to get a thorough review. And they are finding a host of minor issues.

Sometimes I think they brought Btrfs into the main kernel too early, and should have allowed themselves time to work out at least the know wrinkles (such as the ability to robustly detect a full disk). On the other hand, Btrfs has benefited from the review of some really high-powered kernel developers who probably wouldn't have looked at Btrfs too hard if it wasn't in the main kernel.

I guess my overall assessment hasn't changed that much. Btrfs is usable right now, but Btrfs is nowhere near the level of stability that ext4 had when they switched from ext4-dev to stable ext4. There is still a high traffic of patches and fixes, and there are major new features they would like to implement (such as RAID5/6) which have the potential to be disruptive.

I've been personally impressed by the things that Btrfs does well. But I'm eagerly watching the feedback of more knowledgeable people on the possible shortcomings of Btrfs.
Back to top
View user's profile Send private message
ToeiRei
Veteran
Veteran


Joined: 03 Jan 2005
Posts: 1191
Location: Austria

PostPosted: Sun Jun 06, 2010 10:34 am    Post subject: Reply with quote

You should keep in mind that there are a ton of patches sent in just to get other people's opinion about. I'm still following the discussions and do my patch testing. But honestly, there are many people who cannot act as full time developers...
_________________
Please stand by - The mailer daemon is busy burning your messages in hell...
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jun 10, 2010 1:03 am    Post subject: Reply with quote

The response to Edwards's post never came! Interpret that as you will.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Thu Jun 10, 2010 9:32 am    Post subject: Reply with quote

devsk wrote:
The response to Edwards's post never came! Interpret that as you will.


reaction: fail :wink:

why is everyone always feeling that s/he's personally attacked when constructive criticism comes up & digs one's heels in ?
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jun 10, 2010 2:20 pm    Post subject: Reply with quote

kernelOfTruth wrote:
devsk wrote:
The response to Edwards's post never came! Interpret that as you will.


reaction: fail :wink:

why is everyone always feeling that s/he's personally attacked when constructive criticism comes up & digs one's heels in ?
Exactly!
Back to top
View user's profile Send private message
Kingoftherings
Guru
Guru


Joined: 04 May 2008
Posts: 328

PostPosted: Fri Jun 11, 2010 11:29 pm    Post subject: Reply with quote

I'm experiencing a btrfs bug that I've only seen triggered by an emerge. The output gets dumped to my virtual terminal, and I don't know how to retrieve it, as it's not part of stdout or stderr. Within 30 seconds of the bug happening, the entire system locks up.

I'm using linux-2.6.35-rc amd64. I've seen two packages trigger this bug: glibc-2.11.2, and phoronix-test-suite-2.6.1.
I'm recompiling my kernel with some debugging stuff, so I can get some meaningful output. I just don't know how to get that output.
Back to top
View user's profile Send private message
cach0rr0
Bodhisattva
Bodhisattva


Joined: 13 Nov 2008
Posts: 4123
Location: Houston, Republic of Texas

PostPosted: Sat Jun 12, 2010 6:25 am    Post subject: Reply with quote

welp. shit. was going to rebuild this box in the near future anyway, guess it's time to cleanup some stuff on the trusty 'ol backup drive, and move back to XFS/reiser3
Back to top
View user's profile Send private message
petrjanda
Veteran
Veteran


Joined: 05 Sep 2003
Posts: 1557
Location: Brno, Czech Republic

PostPosted: Mon Jun 14, 2010 12:32 am    Post subject: Reply with quote

cach0rr0 wrote:
welp. shit. was going to rebuild this box in the near future anyway, guess it's time to cleanup some stuff on the trusty 'ol backup drive, and move back to XFS/reiser3


Guys, haven't you learned yet through the years of buggy linux file systems, not to use them on anything that you want to keep?
_________________
There is, a not-born, a not-become, a not-made, a not-compounded. If that unborn, not-become, not-made, not-compounded were not, there would be no escape from this here that is born, become, made and compounded. - Gautama Siddharta
Back to top
View user's profile Send private message
cach0rr0
Bodhisattva
Bodhisattva


Joined: 13 Nov 2008
Posts: 4123
Location: Houston, Republic of Texas

PostPosted: Mon Jun 14, 2010 3:41 am    Post subject: Reply with quote

petrjanda wrote:
cach0rr0 wrote:
welp. shit. was going to rebuild this box in the near future anyway, guess it's time to cleanup some stuff on the trusty 'ol backup drive, and move back to XFS/reiser3


Guys, haven't you learned yet through the years of buggy linux file systems, not to use them on anything that you want to keep?


What I've learned is, keep backups, have an exit strategy. And I do have backups, as well an exit strategy.

Now having said that I thought we were to the point with BTRFS where there might be a teething issue surfacing here and there, but only with some of the more advanced features, and nothing that would really potentially impact me personally.

But well, I was wrong - wrong but not stupid, hence the backups and exit strategy.

Thing is, for a good long while I used a mix of reiser3 and XFS (depending on the purpose of the partition/drive). With the BKL issues in reiser3 found on recent kernels, with ext4 still having non-trivial issues with data loss (and in my own testing that may well contradict proper benchmarks, shit performance), I needed to find a replacement for reiser3. I figured now was as good a time as any to look at BTRFS.

I think it may well be time to go back to XFS across the board. Its performance dealing with large sums of small files isn't really *that* bad that I'm going to notice. Who knows, by the time I'm ready to look at BTRFS again, we may well already have native ZFS. I'm not that fussed either way, as I said, I have backups.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash
Back to top
View user's profile Send private message
petrjanda
Veteran
Veteran


Joined: 05 Sep 2003
Posts: 1557
Location: Brno, Czech Republic

PostPosted: Mon Jun 14, 2010 4:34 am    Post subject: Reply with quote

cach0rr0 wrote:

I think it may well be time to go back to XFS across the board. Its performance dealing with large sums of small files isn't really *that* bad that I'm going to notice. Who knows, by the time I'm ready to look at BTRFS again, we may well already have native ZFS. I'm not that fussed either way, as I said, I have backups.


Yeah but ZFS won't be in vanilla linux kernel. Personally I'd stick to ext3... all these new file systems seem to be half baked with little new functionality that should be in a modern file system, and its taking years to get "stable". ie. file system based RAID. Compression sure, encryption sure but volume management and RAID is best left to software/hardware outside of filesystem. Just my opinion, feel free to disagree.
_________________
There is, a not-born, a not-become, a not-made, a not-compounded. If that unborn, not-become, not-made, not-compounded were not, there would be no escape from this here that is born, become, made and compounded. - Gautama Siddharta
Back to top
View user's profile Send private message
cach0rr0
Bodhisattva
Bodhisattva


Joined: 13 Nov 2008
Posts: 4123
Location: Houston, Republic of Texas

PostPosted: Mon Jun 14, 2010 4:55 am    Post subject: Reply with quote

petrjanda wrote:

Yeah but ZFS won't be in vanilla linux kernel. Personally I'd stick to ext3... all these new file systems seem to be half baked with little new functionality that should be in a modern file system, and its taking years to get "stable". ie. file system based RAID. Compression sure, encryption sure but volume management and RAID is best left to software/hardware outside of filesystem. Just my opinion, feel free to disagree.


RE: ZFS, I thought there were talks of there being some agreement with regards to licensing, which was the only thing keeping it out in the first place? Plus there is already movement, e.g. http://wiki.github.com/behlendorf/zfs/

And I do use ext3 for root always. /var and /usr I had historically used reiser3, and XFS for /home. XFS is very mature at this stage, so I will go back to it I think.

I don't know, we will see if anyone ever comments on Edward's post. If there is significant movement in getting the issue fixed, and that movement doesn't require a wholesale redesign that would require people to torch existing BTRFS volumes, I won't be so quick to leave it. But it does look a bit daunting.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Thu Jun 17, 2010 10:22 pm    Post subject: Reply with quote

Is anyone here subscribed to that mailing list? Can you please bump that email from Edward?

The response to that email is critical in two ways:

1. Are the BTRFS developers able to tackle other FS developers with good technical responses?
2. Is there a solution to that issue which can be implemented in a forward compatible manner?

After observing the mailing list and reading some of the "what a good developer should not say" threads, I have doubts about both of these. Otherwise, we would be seeing a response from Chris Mason (who seems to be delegating a lot of stuff to Josef/Zheng/Dan) himself.
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Fri Jun 18, 2010 1:54 pm    Post subject: Reply with quote

Since devsk's post yesterday, there have been a few replies to that thread (and I'm sure more will soon come).

The first reply was a gentle request for a bump.

Then Edward unleashed a ferocious broadside against btrfs with a follow-up response: Btrfs: broken file system design (was Unbound(?) internal fragmentation in Btrfs)

I'm sure that one will draw a response or two.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Fri Jun 18, 2010 3:28 pm    Post subject: Reply with quote

Although I trust Edward technically (he is done some very good work on R4 and understands filesystems well), I think he is a lousy communicator. I don't want that discussion to turn into flames and be ignored. I want to see a good battle between two filesystem developers (and hopefully learn something about internals of the FS on the way).

More than anything, I want to see BTRFS come out as winner. But until this problem is resolved, that's not going to happen. I don't want to see ENOSPC when I have 35% FS free.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Jun 18, 2010 5:27 pm    Post subject: Reply with quote

thank God the discussion continues constructively ^^
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
cach0rr0
Bodhisattva
Bodhisattva


Joined: 13 Nov 2008
Posts: 4123
Location: Houston, Republic of Texas

PostPosted: Fri Jun 18, 2010 8:35 pm    Post subject: Reply with quote

kernelOfTruth wrote:
thank God the discussion continues constructively ^^


and I understand maybe 5% of it. It's certainly...well, it's fuel for learning/reading/googling.
Back to top
View user's profile Send private message
low
n00b
n00b


Joined: 25 Mar 2009
Posts: 18
Location: Maryland

PostPosted: Sun Jun 20, 2010 1:41 am    Post subject: Reply with quote

For initial formatting, would the kernel I use in the live environment where I create the file system matter more/as much/less than the kernel I use on that file system? In other words, should I download and burn livecd's that have the 2.6.35 kernel so I can take advantage of newer patches (even though it is still only git) or will sysreccd's 2.6.34 kernel be ok for creating btrfs even though I'll be using the 2.6.35 kernel (possibly zen) as soon as I get gentoo up and running? How much does the physical file system structure change between kernel versions compared to the file system processes?
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Jun 20, 2010 9:58 am    Post subject: Reply with quote

low wrote:
For initial formatting, would the kernel I use in the live environment where I create the file system matter more/as much/less than the kernel I use on that file system? In other words, should I download and burn livecd's that have the 2.6.35 kernel so I can take advantage of newer patches (even though it is still only git) or will sysreccd's 2.6.34 kernel be ok for creating btrfs even though I'll be using the 2.6.35 kernel (possibly zen) as soon as I get gentoo up and running? How much does the physical file system structure change between kernel versions compared to the file system processes?


according to the mailing list it (the structure / filesystem format) is finalized but I wouldn't rely on that too strongly: they're currently discussing those issues Edward mentioned and it *could* (but is unlikely) that the format changes
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Mon Jun 21, 2010 7:10 pm    Post subject: Reply with quote

Looks like a patch resulted from the discussion. Good!
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon Jun 21, 2010 9:00 pm    Post subject: Reply with quote

devsk wrote:
Looks like a patch resulted from the discussion. Good!


indeed !

nice :P

now let's see how this improves the current situation
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Wed Jun 30, 2010 2:47 am    Post subject: Reply with quote

My latest update: https://forums.gentoo.org/viewtopic-t-834065.html

I gave up on BTRFS. Back to ext4. I got rid of two things by doing that: 1. 20000 kernel threads which BTRFS started, 2. 1-2MB of data written per minute in an idle system.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2, 3 ... 9, 10, 11  Next
Page 1 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum