Multi-device Btrfs root fails to mount without initrd

the8lack8ox

I know this issue has been touched on briefly in these forums, but I'd like to revive it.

I have a RAID10 array of six HDDs containing a Btrfs filesystem as the third partition on each. In this Btrfs filesystem, are three subvolumes (root, boot and home). The system boots via GRUB2 (the only choice it would seem for people with multi-device Btrfs filesystems using LZO compression, unfortunately). Right now, I have an initrd built by dracut that takes care of the device scanning required to build the Btrfs array, which works fine.

Now I'm looking to get rid of the annoying initrd image, whose sole purpose is to perform that one Btrfs scan. It's not that it's causing me any trouble, really, my OCD is just getting the better of me here. I've seen that to make the magic happen, all you need to do, supposedly, is add a device option for each partition in the array via the rootflags kernel parameter. I also added a rootfstype parameter, just for kicks. So this gives a kernel command line like this:

/boot/kernel-3.3.5-gentoo root=/dev/sda3 ro rootflags=device=/dev/sda3,device=/dev/sdb3,device=/dev/sdc3,device=/dev/sdd3,device=/dev/sde3,device=/dev/sdf3,subvol=root rootfstype=btrfs quiet

And I reboot and... my root filesystem fails to mount. The first time it's supposed to be mounted, I mean. If I take off the quiet, the kernel reports that I have 6 disks to choose from containing 3 partitions each. Well, great! That's exactly what it should say! So, why won't my root filesystem mount? Any ideas? I'm lost and don't know how to get any more info out of the kernel.

massimo · Veteran Joined: 22 Jun 2003 Posts: 1226

Please refer to [1]. You should have put those parameters/options into the corresponding fstab entry and not into boot line.

[1] btrfs wiki
_________________
Hello 911? How are you?

the8lack8ox · Posted: Tue May 15, 2012 10:59 am Post subject:

I've done that, and it doesn't matter anyway. The problem occurs when the root FS is first being mounted (fstab isn't accessible yet). The kernel needs to be informed of the geometry of the RAID array, which should be taken care of the device options on rootflags, but it's almost like the btrfs mounter is ignoring those options or not seeing them or something.

massimo · Veteran Joined: 22 Jun 2003 Posts: 1226

Try it with subvol=@root (note the @ which is missing in your case).
_________________
Hello 911? How are you?

the8lack8ox · Posted: Tue May 15, 2012 11:53 am Post subject:

Still no good with the subvol=@root.

Starting to look thru kernel code. Whee!

py-ro · Posted: Tue May 15, 2012 12:59 pm Post subject:

Kernel can't detect multi volume btrfs (and as far as i know, nobody wants to add it). You need an initrd issuing the btrfs scan, if u want to have / on multivolume btrfs.

the8lack8ox · Posted: Tue May 15, 2012 1:19 pm Post subject:

I'm getting closer to the source of the problem. I added a bunch of printk's to my kernel to get some decent error reportage. It looks like, oddly, that the device files /dev/sda3, /dev/sdb3, and so on aren't there when the btrfs_mount is called. Right now I'm looking at btrfs_scan_one_device, which calls blkdev_get_by_path and blkdev_get_by_path returns an error.

ulenrich · Veteran Joined: 10 Oct 2010 Posts: 1413

Just a question as dracut should do exactly that:
dependency solving

Do you want to develop an alternative to dracut?
_________________
ultra left enrich

py-ro · Posted: Tue May 15, 2012 2:21 pm Post subject:

The kernel folk don't want this autodetectin in kernel, that was what i tried to say. It is the same reason why metadata > 0.9 are not recognized.

the8lack8ox · Posted: Tue May 15, 2012 4:35 pm Post subject:

The scan function is written in the kernel already as btrfs_scan_one_device in fs/btrfs/volumes.c. When you run btrfs device scan without arguments, all it does is run that routine against all possible devices through an ioctl call.

All I want is the device option to work at boot, so I don't have to use the buzz-saw that is initrd to mount my root filesystem. Easy enough one would think. I'm not auto-detecting anything. In fact, I'm just trying to friggin' tell it what devices my filesystem is on (but it doesn't want to listen, sadly).

It is clear to me now that /dev is not fully populated at the time the root filesystem is being mounted. Interestingly, the device being mounted to / is something called /dev/root, and the device nodes /dev/sda3, /dev/sdb3, and so on are not present at that point in time. This document is rather interesting and enlightening of the situation.

I was hoping that the fairly new CONFIG_DEVTMPFS_MOUNT, which "Automount[s] devtmpfs at /dev, after the kernel mounted the rootfs", would give access to those devices when attempting to mount the real root fs. And apparently, it does not, or is not ready to, or whatever. (What is that option really good for anyway other than causing boot failures when you upgrade udev! Bah I say!)

I notice that OpenRC/udev do this thing called "populating /dev though uevents" or something like that. Maybe it doesn't do anything at all; I don't know yet. Anyway, the question becomes if /dev really is mounted right after rootfs is mounted, how do I populate it so I can use it to mount my real root. I guess in the worse case I have to create an initramfs with the device nodes already there. I thought the whole idea of that config was to keep the treadmill going away from the initrd/initramfs stuff.

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6597

Try enabling CONFIG_SCSI_WAIT_SCAN.

the8lack8ox · Posted: Tue May 15, 2012 7:00 pm Post subject:

Bad news. The / filesystem gets mounted before devtmpfs at /dev, and the kernel figures out what is pointed to by the root= parameter separately from everything else. It pretty much goes like this:

Make /dev/root
Mount /dev/root to /root
Mount devtmpfs to /root/dev
Mount /root to /

At least, that's what I can gather from the final few lines of init/do_mounts.c. Hence, the device options in rootflags don't work because the device nodes don't exist yet. It looks like getting that to work would require invasive changes that are not too pleasant.

That's enough for me today. I've been awake for 42 hours straight. I think I'll sleep now.

the8lack8ox · Posted: Wed May 16, 2012 2:04 am Post subject:

At last! Success!

I've hacked my kernel just a tad bit to mount a devtmpfs before attempting to mount the root filesystem. Now I can mount my multi-device Btrfs root filesystem without using an initramfs/initrd! Here is my patch:

s4e8 · Guru Joined: 29 Jul 2006 Posts: 309

You don't need the patch, instead embeded initramfs contains /dev/sdX things plus /dev/console.

shimbob · n00b Joined: 13 Sep 2003 Posts: 61

Nice, good job. I'm investigating putting / on a multi-device btrfs as well. Glad it's possible.

Dark_Ebola · n00b Joined: 28 Dec 2004 Posts: 28

budee · n00b Joined: 16 Apr 2005 Posts: 8

Thank you for the patch the8lack8ox, you are awesome! Unfortunately it didn't apply when I wanted to update from kernel 3.6 to 3.7, so I went ahead to the newest 3.8 and modified it to work with that. Below is my patch, which is working fine for me with 3.8.2.

aventrax · n00b Joined: 01 Jul 2013 Posts: 1

Hello, I found this thread very interesting.

I've got a raid0 btrfs array on three disks and I'm trying to boot it without the initrd.
I applied the devtmpfs patch to mount /dev/ earlier, and I added the complete rootflags specifying the root device composing my root partition..

..without success :/

I'm asking, is it possible to do without the initrd? This thread stops after a while and I'm not sure if somebody has successfully booted the root filesystem on a btrfs raid...

Thanks
M

gus.j.power · n00b Joined: 17 Nov 2010 Posts: 12 Location: London

I had exactly the same issue trying to boot of a raid1 btrfs drive pair. I applied budee's patch above against 3.14.14 and I'm up and running without needing an initrd.

+1 !
_________________
--with-${relevantQuote}

SharkWipf · n00b Joined: 13 Jul 2012 Posts: 12

Had the same issue, managed to get it working thanks to this patch.
Had to modify it a little again to get it to play nice with 4.2, if anyone still runs into this issue, here's the modified patch:

s4e8 · Guru Joined: 29 Jul 2006 Posts: 309

the simplest way is embed required dev-node into rootfs.
(find /dev -name "sd*"; echo /dev/console; echo /root) | cpio -H newc -o > dev.cpio
then put dev.cpio into CONFIG_INITRAMFS_SOURCE

SharkWipf · n00b Joined: 13 Jul 2012 Posts: 12

Roman_Gruber · Posted: Thu Oct 29, 2015 10:41 pm Post subject:

are you sure you can boot such a setup without an initramfs?

one option is to embed the initramfs in the kernel. so you do not see it in your bootloader. but it's still there

SharkWipf · n00b Joined: 13 Jul 2012 Posts: 12

It is possible to boot initramfs-less according to the btrfs documentation, however the patch mentioned earlier is required to actually make it work kernel-side.
With said patch it works, however updating the patch with each kernel update is a bit of a bother.
I'm currently playing with Dracut to generate an initramfs which I will then either embed into the kernel or package with the --uefi flag.
Sadly Dracut's --uefi flag requires a systemd EFI stub file, so I'm gonna have to either rip that from some other box somewhere or stick with embedding.

It's a bit annoying that the extra complexity of an initramfs is required for something that also works when adding 1 line of code to the kernel though (although I must admit I don't know the other implications of that change).

bradskins · n00b Joined: 08 Mar 2016 Posts: 1

has anyone gotten this to work with 4.4.0? I have it working fine on 4.3.3.