Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Create a 'cold' grub install and /boot for added protection
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Tue Jun 14, 2022 8:29 pm    Post subject: Reply with quote

I need to get this straight before I start messing around with my system.

I use BIOS boot with gpt and have reserved/protected space before the first partition.

Code:

Device         Start        End    Sectors   Size Type
/dev/sda1       2048       6143       4096     2M BIOS boot
/dev/sda2       6144     530431     524288   256M Microsoft basic data
/dev/sda3     530432   13113343   12582912     6G Microsoft basic data
/dev/sda4   13113344  640251903  627138560   299G Microsoft basic data
/dev/sda5  640251904 1953514927 1313263024 626.2G Microsoft basic data


I get that regardless if raid or some other redundant boot setup that I need to install grub to every drive I would want to be able to boot from. Using the BIOS I would instruct which drive to boot/load grub from. /dev/sda2 is my /boot which I will be wanting to switch to a mirrored raid with sdb2 sdc2. /dev/sda3 is swap, /dev/sda4 and /dev/sda5 are part separate ZFS raidz pools for both / and a general data storage pool.

This is how I think the boot process works...

  • system BIOS boots the boot loader from the specified device, not sure if this is from the protective area before the BIOS boot partition or from the BIOS boot partition (maybe not too important for this....grub is started)
  • with grub started it begins the boot process first locating where /boot is. I guess at this stage is when applicable grub modules are loaded to be able to find boot...i.e. ext2 raid etc
  • with boot found it then loads the kernel and initramfs as instructed in /boot/grub/grub.cfg and passes to the kernel the value of root (os root) which in my case is the rpool zfs pool
  • kernel is loaded and switch to root made

Based on the above, if I put /boot on raid1 I would think then grub needs to be raid aware and of course the kernel/user space needs to be raid aware as well (not to boot but to make changes to /boot while mounted as a raid device). BIOS does not need to be raid aware...it just needs to boot from a device that has a raid aware grub installed to it. I guess NeddySeagoon you are suggesting using .90 metadata so that each disk can be booted from by grub without grub being raid aware while in the OS all information on the drives will be maintained by way of raid? In this scenario I don't know what I would use for --set root that would work across all three drives...

I read online that grub2 is able to assemble raids using metadata 1.x using rmdaid1x. So I am thinking that I can have grub assemble the raid even in a degraded state and boot just fine no? I feel I am missing something in this whole process....
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 50694
Location: 56N 3W

PostPosted: Tue Jun 14, 2022 9:10 pm    Post subject: Reply with quote

mondjef,

Lets go into a little more detail about the on disk data layout an boot process.
You have a MBR at logical block address 0. That all the BIOS can read.

LBA 0 contains the first stage boot loader and the protective MSD partition table.
LBA 1 is the start of GPT. There is no free space before the first partition, so grub cannot use it.
Instead, you have the 2M BIOS Boot partition.
Your /boot filesystem starts in block zero of the /boot partition. Choosing raid metadata version 0.90 does not change that. The raid data goes at the end of the boot partition,

When you install grub to the MBR, its in three pieces.
The stage 1 boot loader goes into LBA 0. It's at most 446 bytes, so it can't do very much.
The stage 1.5 goes into the BIOS Boot partition
Stage 2 goes into the /boot filesystem.

At boot time, the BIOS loads LBA 0 into RAM and jumps to its start address, so that it executes.
It loads stage 1.5 by making BIOS calls. When stage 1.5 is loaded stage 0 jumps to the stage 1.5 start address, is that it can execute.
Stage 1.5 can read exactly one filesystem. It loads stage 2 from /boot, then jumps to it when its loaded.

Stage 2 shows you the boot menu. Grub does its thing and jumps to the kernel start address.

Grub2 is raid aware but with version 0.90 raid metadata it won''t matter.
The kernel needs raid1 support but it hides the raid from the rest of userspace. You will need the mdadm tool to create and manage the raid.

The /boot raid1 need not be assembled for reading. While its not assembled, its like three copies of the same thing.
However, grub and the kernels it loads do not have to be on the same partition.
If the kernels are on a say a raid5, then grub needs to assemble the raid to read the kernels.

Here's my arm64 server grub.cfg It does what I'm trying to describe but its EFI.
There is no stage 1 or stage 1.5.
Everything is raid1 but only boot is metadata 0.9

Code:
# mdadm -E /dev/sda[123]
/dev/sda1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a25b05eb:3db18cbe:afb9312b:d1d97546
  Creation Time : Mon Jan 17 14:54:27 2022
     Raid Level : raid1
  Used Dev Size : 249792 (243.94 MiB 255.79 MB)
     Array Size : 249792 (243.94 MiB 255.79 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Tue Jun 14 22:03:02 2022
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : cf5cd37f - correct
         Events : 57


      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       0        0        1      faulty removed
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : de8f2cbc:17ca3275:0b69db3c:b9f91a6b
           Name : moriarty:1
  Creation Time : Mon Jan 17 14:56:05 2022
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 124430336 sectors (59.33 GiB 63.71 GB)
     Array Size : 62215168 KiB (59.33 GiB 63.71 GB)
    Data Offset : 67584 sectors
   Super Offset : 8 sectors
   Unused Space : before=67432 sectors, after=0 sectors
          State : clean
    Device UUID : 28ba8d69:2f809ea9:0cdd60a9:973efd4d

    Update Time : Tue Jun 14 22:04:03 2022
  Bad Block Log : 512 entries available at offset 136 sectors
       Checksum : 435e7fa9 - correct
         Events : 13965


   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : a3aab047:413ed52d:b15158fc:cdb637ef
           Name : moriarty:2
  Creation Time : Mon Jan 17 14:56:41 2022
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1828259840 sectors (871.78 GiB 936.07 GB)
     Array Size : 914129920 KiB (871.78 GiB 936.07 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : clean
    Device UUID : 60a95480:b41030a9:006ecf7d:a04860a2

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun 14 21:46:55 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : d5725273 - correct
         Events : 458053


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing, 'R' == replacing)


I had both drives fail a short time apart, so its on one old drive right now.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
GDH-gentoo
l33t
l33t


Joined: 20 Jul 2019
Posts: 887
Location: South America

PostPosted: Tue Jun 14, 2022 9:17 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Here's my arm64 server grub.cfg It does what I'm trying to describe but its EFI.

In this server, the partitions that make up the RAID device where /boot is are also the ESPs of each drive, or are the ESPs different partitions?
_________________
NeddySeagoon wrote:
I'm not a witch, I'm a retired electronics engineer :)
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 15, 2022 12:59 am    Post subject: Reply with quote

Thanks neddyseagoon for that explanation!

So using raid1 with .90 metadata negates dealing with having grub raid aware.

The part I'm confused on now is the grub configuration part to allow me boot from any one of the drives. Using raid1 is basically the some thing I was doing before when I was dd'ing the /boot partition from sda2 to the /boot partitions of the other drives...other than of course the added convenience that it is easier to manage from the os side.

What should grub root be set to and what uuids would I expect to see referenced in the grub config? Because grub will not be assembling any raid I would assume grub root would not be referencing anything like md0. If it should be (hd0,1) how can I boot from second drive? Or if select in bios to boot from second drive it becomes first drive?
Back to top
View user's profile Send private message
GDH-gentoo
l33t
l33t


Joined: 20 Jul 2019
Posts: 887
Location: South America

PostPosted: Wed Jun 15, 2022 1:05 pm    Post subject: Reply with quote

mondjef wrote:
What should grub root be set to and what uuids would I expect to see referenced in the grub config?
The UUID of the ext2 filesystem of the RAID device. That's where the kernel that GRUB is asked to boot, and its accompanying initramfs, will be found.

mondjef wrote:
Because grub will not be assembling any raid [...]
Yes, it will. That's the point. GRUB will call the RAID device (mduuid/UUID) or something like that.

mondjef wrote:
So using raid1 with .90 metadata negates dealing with having grub raid aware.
I believe that metadata format 0.90 is relevant for NeddySeagoon's server, because it uses UEFI, the partitions that make up the RAID device are also the EFI System Partitions (ESP) of each drive, and UEFI firmware needs to access files in the ESP's filesystem. I don't think it matters for your case (BIOS booting), because the BIOS firmware would just execute code in the MBR. That's why I asked NeddySeagoon.
_________________
NeddySeagoon wrote:
I'm not a witch, I'm a retired electronics engineer :)
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Tue Jun 21, 2022 4:18 pm    Post subject: Reply with quote

took a stab at this last weekend without success....here is a re-account of my steps and the results.

Current regular boot device is /dev/sda and /boot partition is /dev/sda2. /dev/sdb and /dev/sdc are partition exactly the same as /dev/sda and each have second partition intended for /boot as well. Overall goal is to have sda2, sdb2, and sdc2 in a raid1 array that grub will boot from and load kernel and initrd from and where os will use zfs as root. Initrd handles the / zfs part, /boot is only mounted when needed (i.e. kernel upgrades and/or grub changes).

  • used sgdisk to changed partition type of /dev/sdb2 to FD (Linux Raid). Will do this to the other partitions later.
  • created a degraded raid with just sdb2 using metadata 1.2 (mdadm --create --verbose /dev/md0 --level=1 --raid-devices=3 /dev/sdb2 missing missing)
  • format raid with filesystem (mkfs.ext2 /dev/md0)
  • mounted raid (mount /dev/md0 /mnt/boot)
  • copied /boot from /dev/sda2 to raid (rsync -aH -vih --progress --delete /boot/ /mnt/boot/)
  • updated /etc/fstab to mount raid on /boot (UUID=63d6be6c-b26d-46f7-ab86-139bcf72b48c /boot ext2 noauto,noatime 1 2)
  • re-emerged grub (read somewhere that grub needs to be re-ermeged once mdadm is installed on system...so did for good measure)
  • grub-install /dev/sdb so that core image will be updated with needed modules for mount raid that /boot is on
  • grub-mkconfig to produce a new grub.cfg


Where things seem to go sideways is at grub-install /dev/sdb, below is the output...
Code:

Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.


Googled the above warning and there were a number of threads saying this can be related to partition changes that the kernel is unaware of it.....I did change the partition type so thought ok this reasonable to see and so the simple fix is to reboot which I did (from sda) and retried but just get the same warnings. There were some threads that indicated that the person just ignored these warnings and their system was just fine so I tried push onward.

Code:
lsblk -o NAME,UUID
NAME      UUID
sda
├─sda1
├─sda2    0e3fb725-b0f5-4b0f-a8fe-b61cb9e2fbb1
├─sda3    da5c73e5-a0a5-475b-8d0c-acf288910cfe
├─sda4    12325111364236511523
└─sda5    12351372078106659291
sdb
├─sdb1
├─sdb2    9d216ec3-9689-0076-3999-6ea828edcd0d
│ └─md117 63d6be6c-b26d-46f7-ab86-139bcf72b48c
├─sdb3    146589f3-3a77-4e47-8e7d-18dc308e39c8
├─sdb4    12325111364236511523
└─sdb5    12351372078106659291
sdc
├─sdc1
├─sdc2
├─sdc3
├─sdc4    12325111364236511523
└─sdc5    12351372078106659291


Note: I created the array as /dev/md0 but it now switched to /dev/md117, a bit annoying but I did not reference /dev/md0 anywhere in configs so should not have caused any issues.

mdadm -E /dev/sdb2
Code:

/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9d216ec3:96890076:39996ea8:28edcd0d
           Name : rivermistbeast:0  (local to host rivermistbeast)
  Creation Time : Fri Jun 17 19:52:04 2022
     Raid Level : raid1
   Raid Devices : 3

 Avail Dev Size : 522240 (255.00 MiB 267.39 MB)
     Array Size : 261120 (255.00 MiB 267.39 MB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : df72273b:3ccf7a66:6be9a76c:9a365d23

    Update Time : Mon Jun 20 21:44:14 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : fa72bf7a - correct
         Events : 74


   Device Role : Active device 0
   Array State : A.. ('A' == active, '.' == missing, 'R' == replacing)


Below is my grub.cfg that was produced by the grub-mkconfig command. Seems to identify my raid device correctly however there are some questionable things... for example why are there multiple ismod part_gpt entries above prior to the first menu entry and why is there now ismod zfs when my systems uses an initrd to handle this part? The zfs modules was not there with the same setup minus the raid part (i.e. my grub.cfg generated by grub-mkconfig on /dev/sda2 does contain this).

Code:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod part_gpt
insmod part_gpt
insmod zfs
set root='hd0,gpt4'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt4 --hint-efi=hd0,gpt4 --hint-baremetal=ahci0,gpt4  --hint-bios=hd1,gpt4 --hint-efi=hd1,gpt4 --hint-baremetal=ahci1,gpt4  --hint-bios=hd2,gpt4 --hint-efi=hd2,gpt4 --h>
else
  search --no-floppy --fs-uuid --set=root ab0b977c0a728923
fi
    font="/ROOT/gentoo@/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Gentoo GNU/Linux' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-ab0b977c0a728923' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod ext2
        set root='mduuid/9d216ec39689007639996ea828edcd0d'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/9d216ec39689007639996ea828edcd0d'  63d6be6c-b26d-46f7-ab86-139bcf72b48c
        else
          search --no-floppy --fs-uuid --set=root 63d6be6c-b26d-46f7-ab86-139bcf72b48c
        fi
        echo    'Loading Linux 5.15.41-gentoo ...'
        linux   /vmlinuz-5.15.41-gentoo root=rpool/ROOT/gentoo ro triggers=zfs
        echo    'Loading initial ramdisk ...'
        initrd  /initrd-5.15.41-gentoo

}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###


When I try to boot from /dev/sdb (/boot on raid) it fails and drops to the grub rescue prompt. As a test, I added a menue entry to grub.cfg on /dev/sda2 for the raid setup referencing to the mduuid and if I boot from /dev/sda and select this entry I can boot the system. Although I don't think this is a good test as I believe at that point the grub menu is displayed and grub is reading from the grub.cfg on /dev/sda2 but if I select the menu entry for the raid setup would it not be assembling the raid and loading the kernel and initrd from the raid device though?

I am stuck now and not sure where to go from here....
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 50694
Location: 56N 3W

PostPosted: Tue Jun 21, 2022 4:50 pm    Post subject: Reply with quote

mondjef,

grub will assemble your raid set for itself That does not happen until the stage2 is load though.

The BIOS, which is completely unaware of raid and the early stages of grub, also unaware of raid, will not.
Grubs root, where the stage2 is installed and the files you want to boot should be your raid set.

You have to get the stage2 loaded first, so when you install grub to the MBR, you do in one to each drive.
My server with BIOS and GPT still uses grub-legacy to boot, so I've never tried BIOS mint grub2.

The point about raid metadata 0.90 is to keep the filesystem start in the same place, so it can be read by raid aware and non raid aware.
It matters a great deal on EFI and with grub-legacy, which is not raid aware, both of which need to read the filesystem.GDH-gentoo.

GDH-gentoo,

The raid 0.90 set is the EFI partition too.

-- edit --

Code:
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Gentoo GNU/Linux' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-ab0b977c0a728923' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod ext2
        set root='mduuid/9d216ec39689007639996ea828edcd0d'


That looks OK, the gpt module, the miraid1x module and set root mduuid/9d216ec39689007639996ea828edcd0d which is /dev/sdb2

But ... how does grub (any part of it) read this file when it has raid metadata where the filesystem should start?
/dev/sdb2 is a piece of a raid set, as configured there is no filesystem at the start of /dev/sdb2.
I don't know how grub2 loads its stage2 in detail.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Tue Jun 21, 2022 9:17 pm    Post subject: Reply with quote

All I can guess at this point is that for some reason there are modules missing from the grub core image that gets placed in BIOS boot partition that prevents it from assembling the raid to read the /boot and load stage 2 grub?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 50694
Location: 56N 3W

PostPosted: Tue Jun 21, 2022 9:26 pm    Post subject: Reply with quote

mondjef,

Grub2 can't know to assemble the raid until it reads the grub.cfg file, which is on the raid that it doesn't know that it needs to assemble to read the grub.cfg file, if you see what I mean.

Try raid metadata 0.90 on /boot, then if the raid is assembled or not, it won't matter to the boot process.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 22, 2022 1:22 am    Post subject: Reply with quote

NeddySeagoon wrote:
mondjef,

Grub2 can't know to assemble the raid until it reads the grub.cfg file, which is on the raid that it doesn't know that it needs to assemble to read the grub.cfg file, if you see what I mean.

Try raid metadata 0.90 on /boot, then if the raid is assembled or not, it won't matter to the boot process.


I was under the impression that part of the grub-install 1) makes sure the core.img contains the need modules to access boot and 2) some information as to where /boot is. This is why always assumed that many tutorials always gets you to make sure /boot is mounted prior to running grub-install.

So the part I am having difficulty with is realizing what benefits this has over coping /dev/sda2 partition to the other drives to serve as a backup and keeping grub-install on those drives current as well periodically when kernel upgrades are done or boot configurations are made and foregoing any added value the raid setup offers when not actually having grub boot from it and thus be fault tolerant. With doing things manually, in the event /dev/sda becomes unavailable (even in bios) one of the other drives will become the first drive and I should be able to boot from it with no issues still. In the event /dev/sda remains available from bios but is otherwise experiencing issues preventing booting I can simply unplug it and boot again from one of the other drives.

For me, if I can't have grub assemble the raid to read from /boot then the added layer of having the raid in the mix at all is negligible.
Back to top
View user's profile Send private message
GDH-gentoo
l33t
l33t


Joined: 20 Jul 2019
Posts: 887
Location: South America

PostPosted: Wed Jun 22, 2022 3:04 am    Post subject: Reply with quote

mondjef wrote:
  • created a degraded raid with just sdb2 using metadata 1.2 (mdadm --create --verbose /dev/md0 --level=1 --raid-devices=3 /dev/sdb2 missing missing)
mondjef wrote:
Where things seem to go sideways is at grub-install /dev/sdb, below is the output...
Code:

Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.

There is a comment immediately above the code that produces that output:

grub-core/disk/diskfilter.c:
static grub_disk_memberlist_t
grub_diskfilter_memberlist (grub_disk_t disk)
{
  // ...
  for (pv = lv->vg->pvs; pv; pv = pv->next)
    {
      if (!pv->disk)
        {
          /* TRANSLATORS: This message kicks in during the detection of
             which modules needs to be included in core image. This happens
             in the case of degraded RAID and means that autodetection may
             fail to include some of modules. It's an installation time
             message, not runtime message.  */
          grub_util_warn (_("Couldn't find physical volume `%s'."
                            " Some modules may be missing from core image."),
                          pv->name);
          continue;
        }
      // ...
    }
  // ...
}
And you created a degraded RAID...

mondjef wrote:
[...] however there are some questionable things... for example why are there multiple ismod part_gpt entries above prior to the first menu entry and why is there now ismod zfs when my systems uses an initrd to handle this part?
This part:

grub.cfg:
set root='hd0,gpt4'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt4 --hint-efi=hd0,gpt4 --hint-baremetal=ahci0,gpt4  --hint-bios=hd1,gpt4 ...
else
  search --no-floppy --fs-uuid --set=root ab0b977c0a728923
fi
    font="/ROOT/gentoo@/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  ...
fi
loads a font for GRUB's graphic terminal (gfxterm). It wants font unicode.pf2, which apparently is only found in /usr/share/grub... which is on ZFS. I don't know if this is normal or not, on my computer I installed sys-boot/grub with the fonts USE flag unset.

mondjef wrote:
I am stuck now and not sure where to go from here....
Maybe try with an array composed of /dev/sdb2 and /dev/sdc2 instead of a degraded array?

mondjef wrote:
All I can guess at this point is that for some reason there are modules missing from the grub core image that gets placed in BIOS boot partition [...]
We might be able to check. Put the output of grub-install --verbose /dev/sdb in a pastebin site. It is big. In particular, it will be interesting to see the call to grub-mkimage (the internal program that creates /boot/grub/i386-pc/core.img) in that output.

mondjef wrote:
I was under the impression that part of the grub-install 1) makes sure the core.img contains the need modules to access boot and 2) some information as to where /boot is.
I also believe that that's what should happen.
_________________
NeddySeagoon wrote:
I'm not a witch, I'm a retired electronics engineer :)
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 22, 2022 12:31 pm    Post subject: Reply with quote

Not sure how the fonts flag got set for grub as I do not have it set anywhere...not sure if it was there prior to messing around with the raid stuff but for sure those lines related to the font it wants is not in my currently working grub.cfg (non-raid version). I am currently re-emerging grub with the fonts flag unset and will see if that fixes that issue although I would think there would be an easy way to tell grub to not use any special fonts even when the fonts flag is set....

Also, I temporarily changed my mirrored array to an array of only 1 disk (no missing disks) and was able to run grub-install without issue now. Once I get this working I will grow the array to 2 disks and tests and if that works I will extend to the full 3 disks and fully adopt the set up.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 22, 2022 1:46 pm    Post subject: Reply with quote

I had some success with the single disk raid1 (non-degraded mode). I had a real hard time distinguishing which disk identified in the bios mapped to which linux device as they all used the same label in the bios. Ended up modifying the menu entry label to something unique for the grub.cfg file stored on the md0 so that I can be sure that I was booting from the grub that should be raid aware. After re-emerging grub with the fonts flag unset and doing grub-install and grub-mkconfig I noticed two interesting things...
  • no more problematic font trying to be loaded from zfs which is good
  • and the duplicate 'ismod part_gpt' lines I had previously after grub-mkconfig disappeared (not entirely sure if this was a result of unsetting the fonts flag or from getting rid of the degraded raid so that grub-install did not complain)


I currently have libzfs flag set for grub but I am not entirely sure if I really need it as I am not 'booting' from zfs but instead I have an initrd kernel image loaded from /boot (the one now on raid) that takes care of loaded zfs and then loading the kernel and switching root. My kernel line has something similar to 'root=ZFS=rpool/GENTOO/root triggers=zfs' which from my understanding is just passed to the kernel and grub doesn't even need to understand zfs at all or does grub still need libzfs to do something with the zfs options on the kernel line?

Next steps will be to add another disk (grow) to the raid and retest including yanking a drive out to ensure grub/system reboots prior to fully adopting this. Assuming all this works as planned the redundancy part will be fully automated with the exception of doing the grub-install to each disk that is part of the raid so all in all I think my system is fairly fault tolerant when it comes to any potential disk failure. The only thing remaining that as a bit of a weak point is when new zfs modules are emerged I have to keep my initrd kernal image insync with the same versions (especially when pools are upgraded to the newest feature sets) as failure to do so will render my system unbootable and would require a rescue environment with the applicable recent zfs modules to fix. But I have limited these failure points as much as possible at least.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 22, 2022 9:17 pm    Post subject: Reply with quote

test complete!

I added one of the other drives to the raid and did grub-install to that drive as well, rebooted and was able to boot from either drive and grub was able to assemble raid and boot initrd and kernel just fine. I then yanked out one of the drives and rebooted....again was able to boot from remaining drive and grub was able to assemble degraded raid and boot initrd and kernel which then proceeded to boot a degraded system as the zfs pools also had partitions on the drive that I yanked. So all in all a success.

However, what I noticed in the degraded os is that I was unable to mount my /boot as a degraded raid as it complained that it could not find UUID '63d6be6c-b26d-46f7-ab86-139bcf72b48c' which is the UUID I have in my fstab and is what I believe to be the filesystem UUID that is on top of my raid. So I think I have something messed up with my raid in fstab that prevents me from mounting in degraded mode.

Code:
lsblk -o NAME,UUID,MOUNTPOINT
NAME    UUID                                 MOUNTPOINT
sda
├─sda1
├─sda2  9d216ec3-9689-0076-3999-6ea828edcd0d
│ └─md0 63d6be6c-b26d-46f7-ab86-139bcf72b48c /boot
├─sda3  146589f3-3a77-4e47-8e7d-18dc308e39c8 [SWAP]
├─sda4  12325111364236511523
└─sda5  12351372078106659291
sdb
├─sdb1
├─sdb2  9d216ec3-9689-0076-3999-6ea828edcd0d
│ └─md0 63d6be6c-b26d-46f7-ab86-139bcf72b48c /boot
├─sdb3  9563f9ff-c487-47df-afbb-fc7a439d8751 [SWAP]
├─sdb4  12325111364236511523
└─sdb5  12351372078106659291


Any ideas?


Last edited by mondjef on Wed Jun 22, 2022 9:26 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 50694
Location: 56N 3W

PostPosted: Wed Jun 22, 2022 9:24 pm    Post subject: Reply with quote

mondjef,

Is the raid assembled?

In some situations mdadm needs to be encouraged ta assemble a degraded raid.
Grub does not hand over the assembled raid to the kernel, mdadm has to do that once the kernel starts.
Kernel raid auto assemble is for metadata 0.90 raid sets only, so the kernel can't do it.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Wed Jun 22, 2022 9:29 pm    Post subject: Reply with quote

NeddySeagoon wrote:
mondjef,

Is the raid assembled?

In some situations mdadm needs to be encouraged ta assemble a degraded raid.
Grub does not hand over the assembled raid to the kernel, mdadm has to do that once the kernel starts.
Kernel raid auto assemble is for metadata 0.90 raid sets only, so the kernel can't do it.


In the degraded os I did not assemble the raid first....just did the mount /boot and assumed that everything is taken care of :D With a perfectly running raid I am able to do this and mounts just fine...I guess in a degraded state I have to do something extra to force it to assemble?
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Thu Jun 23, 2022 12:37 am    Post subject: Reply with quote

I couldn't seem for the life of me to get my /boot array to assemble in a degraded state after booting my system with a disk missing. I tried everything and when I looked the disk layout everything looked good with /dev/md0 associated with /dev/sda2 (the linux raid).

Finally I found something that worked....

mdadm --stop /dev/md0

then
mdadm -A /dev/md0

and the array would assemble and would indicate it was using 1 of 2 drives to assemble it. After that I am able to mount /boot on the degraded raid array just fine. But why did I have to stop the array first before it would assemble the array in degraded state?
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Thu Jun 23, 2022 1:50 am    Post subject: Reply with quote

There is more going on here...after rebooting with all drives connected again I noticed that the raid was only using a single disk. To correct I had to stop raid and add the missing disk and then assemble raid again.
Going to retest to see it happens again...hopefully this not how things are suppose to happen I.e. when a drive is unavailable it is automatically removed from array and needs to be manually re-added.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Thu Jun 23, 2022 12:07 pm    Post subject: Reply with quote

From what I could find on the internet the behavior of the failed/missing drive being removed from the raid1 array by mdadm is normal and requires it to be manually added back to the array. This happens as the degraded state causes inconsistencies between the drives and the safest thing for mdadm to do is remove the failed drive from the array.

However what I don't understand is why I had to stop the array and re-assemble it in a degraded mode (prior to adding back the failed drive) and why mdadm just doesn't assemble it in a degraded mode (i.e. best effort to bring it back up). Maybe this is specific to a raid1 with only 2 disks that when both disks become available again and there are inconsistencies detected mdadm is unable to know which drive to trust and thus does not do anything automatically to prevent data loss.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 50694
Location: 56N 3W

PostPosted: Thu Jun 23, 2022 4:41 pm    Post subject: Reply with quote

mondjef,

If you mount a degraded raid read/write the event counts for the members will get out of sync.
That's how mdadm detects stale elements at run time

Code:
  sudo mdadm -E /dev/sd[abcd]3 | grep -e sd -e Events
/dev/sda3:
         Events : 72799
/dev/sdb3:
         Events : 72799
/dev/sdc3:
         Events : 72799
/dev/sdd3:
         Events : 72799


mdadm may assemble the degraded raid but not start it. I think there is a start or run command
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Thu Jun 23, 2022 4:48 pm    Post subject: Reply with quote

NeddySeagoon wrote:
mondjef,

If you mount a degraded raid read/write the event counts for the members will get out of sync.
That's how mdadm detects stale elements at run time

Code:
  sudo mdadm -E /dev/sd[abcd]3 | grep -e sd -e Events
/dev/sda3:
         Events : 72799
/dev/sdb3:
         Events : 72799
/dev/sdc3:
         Events : 72799
/dev/sdd3:
         Events : 72799


mdadm may assemble the degraded raid but not start it. I think there is a start or run command


Ah ok...maybe just needed to run it as it was showing but showed as inactive and with only one device even though both devices were available to the system. This weekend I'm going to finalize the switch to raid for /boot
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum