Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Use RAID metadata 1.2 in boot and root partition
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
solamour
Guru
Guru


Joined: 21 Dec 2004
Posts: 498
Location: San Diego, CA

PostPosted: Fri Jul 29, 2011 8:43 pm    Post subject: [Solved] Use RAID metadata 1.2 in boot and root partition Reply with quote

According to "Gentoo Linux x86 with Software Raid and LVM2 Quick Install Guide" (http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml), "/boot" and "/" need to be metadata 0.90.

Quote:
(About 1/3 into the document)

Please bear in mind that the /boot and / mount points should use an older metadata format which can be specified by the --metadata=0.90 flag:

livecd ~ # mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=0.90 /dev/sda1 /dev/sdb1
mdadm: array /dev/md1 started.
livecd ~ # mdadm --create /dev/md3 --level=1 --raid-devices=2 --metadata=0.90 /dev/sda3 /dev/sdb3
mdadm: array /dev/md3 started.
livecd ~ # mdadm --create /dev/md4 --level=0 --raid-devices=2 /dev/sda4 /dev/sdb4
mdadm: array /dev/md4 started.

But I noticed that when I choose RAID in Ubuntu installation, it uses metadata 1.2 even though I create a single mount point "/" without a separate boot partition.

One thing I noticed is that when I use 0.90, I'm able to install GRUB2 with the following commands.

Code:
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sda
grub-install /dev/sdb
grub-install /dev/sdc
...

But when I use 1.2, I always get the following error message.
Code:
(chroot) livecd / # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
/sbin/grub-probe: error: no such disk.

(chroot) livecd / # grub-install /dev/sda
/sbin/grub-probe: error: no such disk.
Auto-detection of a filesystem of /dev/md0 failed.
Please report this together with the output of "/sbin/grub-probe --device-map="/boot/grub/device.map" --target=fs -v /boot/grub" to <bug-grub@gnu.org>

Because I'm not able to install GRUB2 with metadata 1.2, I need to boot from Ubuntu CD's rescue mode and install the boot loader from there.

In Gentoo, the only way I can boot from a single device "/dev/md0" is when the metadata is 0.90; when I use 1.2 (and install GRUB2 with the help of Ubuntu CD), I see a little bit of kernel messages during the boot and then it stops with the following error message.

Code:
"Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)

Here is the relevant section in the GRUB configuration.

Code:
/boot/grub/grub.cfg

menuentry 'Gentoo GNU/Linux, with Linux 2.6.38-gentoo-r6' --class gentoo --class gnu-linux --class gnu --class os {
        load_video
        insmod raid
        insmod raid5rec
        insmod mdraid1x
        insmod part_msdos
        insmod part_msdos
        insmod part_msdos
        insmod ext2
        set root='(md/0)'
        search --no-floppy --fs-uuid --set=root f78170e1-97aa-4d52-a95e-d02f0c4a0046
        echo    'Loading Linux 2.6.38-gentoo-r6 ...'
        linux   /boot/kernel-2.6.38-gentoo-r6 root=/dev/md0 ro
}


I'd like to know what makes Ubuntu be able to use 1.2 with the same disk configuration.
__
sol


Last edited by solamour on Mon Aug 08, 2011 6:30 am; edited 1 time in total
Back to top
View user's profile Send private message
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2381
Location: Wisconsin (USA)

PostPosted: Sat Jul 30, 2011 5:51 pm    Post subject: Reply with quote

I think there's two parts to this. First, ubuntu always uses an initrd so it doesn't depend on the kernel autorecognition of raid devices - it's handled by the normal raid utility after booting the ramdisk. From what I've read in other places, the kernel autorecognition is probably going away rather than getting updated - they're trying to move as much of that stuff out of the kernel as they can.

The second part, is I suspect they've patched grub to allow the newer metadata format. There's no reason you couldn't take ubuntu's grub and build it on gentoo (or even use their binaries like you've tried already). Of course it'd be nice if gentoo incorporated those patches as well.
Back to top
View user's profile Send private message
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2381
Location: Wisconsin (USA)

PostPosted: Sat Jul 30, 2011 5:56 pm    Post subject: Reply with quote

I think there's two parts to this. First, ubuntu always uses an initrd so it doesn't depend on the kernel autorecognition of raid devices - it's handled by the normal raid utility after booting the ramdisk. From what I've read in other places, the kernel autorecognition is probably going away rather than getting updated - they're trying to move as much of that stuff out of the kernel as they can.

The second part, is I suspect they've patched grub to allow the newer metadata format. There's no reason you couldn't take ubuntu's grub and build it on gentoo (or even use their binaries like you've tried already). Of course it'd be nice if gentoo incorporated those patches as well.

I suspect there's probably a functional difference with the new metadata format, so using it is probably desirable if possible (just a suspicion, perhaps it allows more modifications on a live array, like maybe more resizing or conversion options are possible, as there must be some state and probably a data buffer to indicate how far a conversion has progressed and allow its restart in case the system crashes)
Back to top
View user's profile Send private message
solamour
Guru
Guru


Joined: 21 Dec 2004
Posts: 498
Location: San Diego, CA

PostPosted: Sat Jul 30, 2011 9:15 pm    Post subject: Reply with quote

At first I wasn't sure what I was missing in my kernel, so I tried genkernel, but that didn't make any difference; I got the exactly same error message.

Perhaps Ubuntu added something to their GRUB, because when I did "grub-install -v", it returned "1.99~rc1-13ubuntu3" instead of usual "1.99~rc1".

Come to think of it, I wasn't exactly using Ubuntu's GRUB binaries, because I just booted from Ubuntu CD and used their "grub-mkconfig" and "grub-install" to configure and install the bootloader, but all the GRUB-related files in the disk array are still Gentoo's. I'll borrow their files and see what happens.
__
sol
Back to top
View user's profile Send private message
solamour
Guru
Guru


Joined: 21 Dec 2004
Posts: 498
Location: San Diego, CA

PostPosted: Sun Jul 31, 2011 9:43 pm    Post subject: Reply with quote

I downloaded GRUB2 from Ubuntu's repository (http://archive.ubuntu.com/ubuntu/pool/main/g/grub2/grub2_1.99~rc1.orig.tar.gz) and applied their patch (http://archive.ubuntu.com/ubuntu/pool/main/g/grub2/grub2_1.99~rc1-13ubuntu3.diff.gz). And then I configured and install it. I unmerged Gentoo's GRUB for a good measure.

Code:
./configure --prefix=/
...

*******************************************************
GRUB2 will be compiled with following components:
Platform: i386-pc
With devmapper support: Yes
With memory debugging: No
efiemu runtime: No (cannot compile with -m64 -mcmodel=large -mno-red-zone -nostdlib)
grub-mkfont: No (need freetype2 library)
*******************************************************


At least the compilation went through without a problem, but I received the exactly same error message when I tried to install the bootloader.

Code:
(chroot) livecd / # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
/sbin/grub-probe: error: no such disk.

(chroot) livecd # grub-install /dev/sda
/sbin/grub-probe: error: no such disk.
Auto-detection of a filesystem of /dev/md0 failed.
Please report this together with the output of "/sbin/grub-probe --device-map="/boot/grub/device.map" --target=fs -v /boot/grub" to <bug-grub@gnu.org>


Just out of curiosity, I manually updated "device.map".

Code:
/boot/grub/device.map

[Before]
(hd0)   /dev/disk/by-id/ata-VBOX_HARDDISK_VBe0cd6f27-04f3344e
(hd1)   /dev/disk/by-id/ata-VBOX_HARDDISK_VB1c6d27be-7e81d686
(hd2)   /dev/disk/by-id/ata-VBOX_HARDDISK_VB1f6ca124-4af21533

[After]
(hd0)   /dev/md0


When I did that, at least "grub-mkconfig" and "grub-install" part went through without a problem. I thought all is well, but when I rebooted, I was greeted with the following message.

Code:
GRUB loading.
Welcome to GRUB!

error: no such device: a50fe7a6-9d52-4f5c-b43d-4bb3e4c0e987
Entering rescue mode...
grub rescue> _


So... I think I'm still missing something here. Please share your thoughts. Thank you.
__
sol
Back to top
View user's profile Send private message
solamour
Guru
Guru


Joined: 21 Dec 2004
Posts: 498
Location: San Diego, CA

PostPosted: Tue Aug 02, 2011 8:35 pm    Post subject: Reply with quote

Here is the current status on booting from RAID-5.

* The only combination that works without any special treatment is 0.90 + Partition Type as 0xFD + GRUB2 (from Portage). As long as I have MD_RAID456 and BLK_DEV_DM as built-in in the kernel, everything worked.

* When the metadata is 1.2, having a separate boot partition (something simple, such as RAID-1) was the only way to boot. I tried genkernel/initramfs, Ubuntu's GRUB2, voodoo dance, etc, and nothing worked. Considering that Ubuntu is able to do it, I suppose it's doable, but I'm out of clues.


I might come back to it at some other time, but for now, I'm going to stick with 0.90. Thanks everyone.
__
sol
Back to top
View user's profile Send private message
solamour
Guru
Guru


Joined: 21 Dec 2004
Posts: 498
Location: San Diego, CA

PostPosted: Mon Aug 08, 2011 6:30 am    Post subject: Reply with quote

After a lot of fiddling around, I was finally able to boot from RAID-5 + metadata 1.2 + no boot partition. The missing piece was initramfs; basically even though the kernel is loaded successfully, it has no knowledge of "/dev/md0" unless someone else assembles it for the kernel. I mostly followed Initramfs wiki, which is a good read even if you don't deal with RAID-5 and metadata 1.2.

http://en.gentoo-wiki.com/wiki/Initramfs

Code:
#Create a work directory.
mkdir /usr/src/initramfs
cd /usr/src/initramfs

# Create basic directories for initramfs.
mkdir -p bin lib dev etc mnt/root proc root sbin sys

# Copy devices required for booting.
cp -a /dev/{null,console,tty,md0,sd*} /usr/src/initramfs/dev/

# mdadm needs to be built as static, so that we don't need to
# bother with required libraries.
USE="static" emerge -v mdadm
cp -a /sbin/mdadm /usr/src/initramfs/bin/

# Create "init" file.
/usr/src/initramfs/init
   #!/bin/busybox sh
   mount -t proc none /proc
   mount -t sysfs none /sys

   # I named mine as "gentoo:0". Yours might be different.
   mdadm --assemble --scan /dev/md/gentoo:0
   # OR just "mdadm --assemble /dev/md0 /dev/sd[abc]1"

   mount -o ro /dev/md0 /mnt/root
   umount /proc
   umount /sys
   exec switch_root /mnt/root /sbin/init

chmod +x /usr/src/initramfs/init

# Generate initramfs file.
cd /usr/src/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-2.6.39-gentoo-r3.img

# This is optional if you explicitly assemble /dev/md0.
# But it's always a good idea to make things flexible.
mdadm --detail --scan >> /etc/mdadm.conf
cp -a /etc/mdadm.conf /usr/src/initramfs/etc/


At this point, everything is pretty much ready, except (for some unknown reasons) GRUB2 doesn't want to be installed in /dev/sd[abc]1. Just boot from Ubuntu CD, select "Rescue Mode", assemble /dev/md0, and chroot to /dev/md0. "grub-mkconfig" and "grub-install" should work.
__
sol
Back to top
View user's profile Send private message
branzo
n00b
n00b


Joined: 31 Oct 2011
Posts: 1
Location: Treviso, Italy

PostPosted: Mon Oct 31, 2011 5:24 pm    Post subject: Reply with quote

solamour wrote:
After a lot of fiddling around, I was finally able to boot from RAID-5 + metadata 1.2 + no boot partition. The missing piece was initramfs; basically even though the kernel is loaded successfully, it has no knowledge of "/dev/md0" unless someone else assembles it for the kernel. I mostly followed Initramfs wiki, which is a good read even if you don't deal with RAID-5 and metadata 1.2.

http://en.gentoo-wiki.com/wiki/Initramfs

Code:
#Create a work directory.
mkdir /usr/src/initramfs
cd /usr/src/initramfs

# Create basic directories for initramfs.
mkdir -p bin lib dev etc mnt/root proc root sbin sys

# Copy devices required for booting.
cp -a /dev/{null,console,tty,md0,sd*} /usr/src/initramfs/dev/

# mdadm needs to be built as static, so that we don't need to
# bother with required libraries.
USE="static" emerge -v mdadm
cp -a /sbin/mdadm /usr/src/initramfs/bin/

# Create "init" file.
/usr/src/initramfs/init
   #!/bin/busybox sh
   mount -t proc none /proc
   mount -t sysfs none /sys

   # I named mine as "gentoo:0". Yours might be different.
   mdadm --assemble --scan /dev/md/gentoo:0
   # OR just "mdadm --assemble /dev/md0 /dev/sd[abc]1"

   mount -o ro /dev/md0 /mnt/root
   umount /proc
   umount /sys
   exec switch_root /mnt/root /sbin/init

chmod +x /usr/src/initramfs/init

# Generate initramfs file.
cd /usr/src/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-2.6.39-gentoo-r3.img

# This is optional if you explicitly assemble /dev/md0.
# But it's always a good idea to make things flexible.
mdadm --detail --scan >> /etc/mdadm.conf
cp -a /etc/mdadm.conf /usr/src/initramfs/etc/


At this point, everything is pretty much ready, except (for some unknown reasons) GRUB2 doesn't want to be installed in /dev/sd[abc]1. Just boot from Ubuntu CD, select "Rescue Mode", assemble /dev/md0, and chroot to /dev/md0. "grub-mkconfig" and "grub-install" should work.
__
sol


Thanks for this Mini How-To, it really saved me! It works well even with RAID 0.
Anyway, wiki site is not working, at least, for me.

I just wanted to let you know that you forgot to mention that _we_ have to put a statically-compiled copy of busybox inside the initramfs, contact me if you need more information.
Back to top
View user's profile Send private message
thenextdon13
n00b
n00b


Joined: 26 Sep 2012
Posts: 9

PostPosted: Wed Dec 05, 2012 12:04 am    Post subject: Reply with quote

Just a few notes that may help others;

The kernel builtin driver that assembles arrays (md) does not natively support mdadm metadata 1.2 as of kernel 3.6.9 . This is noted in http://www.kernel.org/doc/Documentation/md.txt
Code:

Superblock formats
------------------

The md driver can support a variety of different superblock formats.
Currently, it supports superblock formats "0.90.0" and the "md-1" format
introduced in the 2.5 development series.

The kernel will autodetect which format superblock is being used.

Superblock format '0' is treated differently to others for legacy
reasons - it is the original superblock format.


I could not find any reference anywhere else on the internet about the md-1 format, so i do not know if it associates with the 1.0 mdadm format. This leaves us (as of 3.6.9) not being able to use mdadm 1.2 superblock metadata as a boot or root partition without having help; that is, you must use initrd of some sort to supply the required software (mdadm) and required libraries.

This forces anyone wanting to boot directly from an image and kernel without initrd to use metablock .90, I believe.[/code]
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2459
Location: Germany

PostPosted: Wed Dec 05, 2012 11:40 am    Post subject: Reply with quote

thenextdon13 wrote:
The kernel builtin driver that assembles arrays (md) does not natively support mdadm metadata 1.2 as of kernel 3.6.9 .


The auto asssembly is deprecated in general. That's what initramfs is there for.

The only reason they don't remove it for 0.90 metadata is that they don't break things in the kernel if they can help it.

Quote:
This forces anyone wanting to boot directly from an image and kernel without initrd to use metablock .90, I believe.


Yes, although you have to be aware of the limits and unreliabilities that come with having metadata at the end instead of at the beginning of a partition. Also partitions may not be larger than 2TB for 0.90.


I use 0.90 (or 1.0) for RAID 1 on /boot only - everything else is 1.2 for me.
Back to top
View user's profile Send private message
thenextdon13
n00b
n00b


Joined: 26 Sep 2012
Posts: 9

PostPosted: Wed Dec 05, 2012 7:34 pm    Post subject: Reply with quote

@frostshutz: Thanks for the explanation.

Do you think we could convince the kernel doc team to put some notes in the md.txt file saying that further development isn't going to happen?
I only say this as it seems to be the canonical source for info about the md driver, and as much searching on google as i've done (must be using the wrong keywords) i can't find anything other than forum posts with people stating that there will be no further development for kernel autoassemble/md.

Thanks!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum