[SOLVED] gentoo with root partition on LVM

[Vieri]

Hi,

I followed the Gentoo LVM Wiki, but I guess I've missed something because I can't mount the root partition when booting.

Some partitions are in MDADM RAID1 and the ROOT partition is in LVM mirror RAID1.

These are some of my relevant steps from livecd:

Code: Select all

genkernel --install --mdadm --mdadm-config=/etc/mdadm.conf --lvm initramfs

I install grub with a line like this one:

Code: Select all

linux /${K} root=/dev/ram0 init=/linuxrc ramdisk=8192 rootfstype=ext4 vga=0 domdadm dolvm real_root=/dev/vgroot/root

The installed system boots, but it fails to find /dev/vgroot/root, so I enter the busybox shell.

I see a message that says that I can copy /run/..something to a USB pendrive, but oddly enough I can't seem to mount the device.
Both lsusb and dmesg show that the device is in /dev/sdc1, but mount/dev/sdc1 /mnt/pendrive says "no such file or directory". Both exist so I don't know what's up.

Anyway, the following commands show what I'm expecting:

Code: Select all

lvm lvdisplay
lvm pvs

Specifically, I see /dev/vgroot/root, two mirrored volumes, etc., except for one suspicious line:

Code: Select all

LV Status  Not available

In fact, if I reboot in the livecd again, I can see that the status is "available":

Code: Select all

# lvdisplay
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  --- Logical volume ---
  LV Path                /dev/vgroot/root
  LV Name                root
  VG Name                vgroot
  LV UUID                qgWICf-J4s1-LozA-RnUL-pMBy-iumd-B1daVw
  LV Write Access        read/write
  LV Creation host, time livecd, 2021-01-08 14:20:45 +0000
  LV Status              available
  # open                 0
  LV Size                187.12 GiB
  Current LE             47904
  Mirrored volumes       2
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4

So it seems that I've done something wrong in the initramdisk, right?

What can I try?

[alamahant]

Hi
Did you install lvm in the chroot?
The live cd has it installed
but maybe its missing in the chroot.
Also you have to enable lvm@boot and lvmetad@default.
Also I much prefer dracut to genkernel.
Its easier for beginners.

[Vieri]

Hi,

Yes, I emerged lvm2 in the jailed root environment. I also added LVM to the boot runlevel. I did not add lvmetad.
However, the problem is occurring in the initramfs, and the LVM volume information seems to be there.
It's just that the "status" is "unavailable". It may be simply because the initramdisk (genkernel) does not "activate" something at boot time.

Never tried dracut.
I guess I'll have to try.

Thanks

[NeddySeagoon]

Vieri,

A few things.

There are two ways to do root in a LV on top of raid.
The one I use is to donate the block devices to an mdadm raid set, then donate the raid set to a PV.
This way the underlying raid is not visible to LVM.

The other way in to have LVM do it all. You appear to be doing that as you get

Code: Select all

# lvdisplay 
...
  Mirrored volumes       2 
...

The kernel parameters

Code: Select all

root=/dev/ram0 init=/linuxrc ramdisk=8192 

have been obsolete for a long time.
The initramfs is no longer a ramdrive and the default init= just works.

When you poke about in the initrd shell, you say everything you expect to see is there.
Can you mount /dev/vgroot/root by hand ?
If not, what is the error?
Is there anything in dmesg?

Busybox does not have less. Use

Code: Select all

dmesg | more

as a pager.

[Vieri]

No, /dev/vgroot/root is nowhere to be seen.
However lvm lvdisplay DOES show the "right info" about /dev/vgroot/root except that the LV status is "not available".
I will try to run vgscan and vgchange -a y within that SHELL.

Regarding the grub line, I understand that this would be enough?

Code: Select all

linux /${K} root=/dev/ram0 rootfstype=ext4 vga=0 domdadm dolvm real_root=/dev/vgroot/root

[NeddySeagoon]

Vieri,

Code: Select all

vgchange -ay

should be enough.
In my install, /dev/vg/ are all symlinks. You need something to create symlinks.

What does

Code: Select all

ls /dev/dm-*

tell and/or

Code: Select all

ls -l /dev/mapper/*

You are looking for real block device files.

[Vieri]

If I enter the SHELL when the boot process complains that the root LVM partition is not there, I can run these commands:

Code: Select all

lvm vgscan
lvm vgchange -a y

and that does it. The root partition /dev/vgroot/root is there as well as all the /dev/mapper/* devices.
If I "exit" the SHELL the boot process resumes perfectly, and I can log into my system.

So there's obviously something wrong with the way I created my initramdisk with genkernel.

[Vieri]

What does

Code: Select all

ls /dev/dm-*

tell

Does not exist.

and/or

Code: Select all

ls -l /dev/mapper/*

Just /dev/mapper/control.

As reported in my previous comment, I solve the issue by running vgchange -ay.
However, now I need to understand why genkernel did not generate a "proper" initramdisk.

[NeddySeagoon]

Vieri,

I've not used genkernel, so I can't help further.

[alamahant]

In the beginning genkernel used to wrork for me for booting root on lvm.
At some point it stopped.
Then I switched to dracut.
And have never looked back since.

[gentoo_ram]

I'm using root on LVM with an initramfs created by genkernel.

Make sure you have LVM="yes" in your /etc/genkernel.conf
I also have these set: E2FSPROGS, BUSYBOX, LUKS, MDADM, DISKLABEL

For my boot string I have:
linux /vmlinuz-5.10.2-gentoo root=/dev/mapper/vg0-root ro rootfstype=ext4 dolvm
echo 'Loading initial ramdisk ...'
initrd /intel-uc.img /initramfs-5.10.2-gentoo.img

The intel-uc.img is for Intel firmware updates. You may or may not need those.

Then 'genkernel initramfs' and 'grub-mkconfig -o /boot/grub/grub.cfg'.

[skellr]

Yeah, Genkernel initramfs is _deceptively_ easy to use. You don't need to add extra kernel parameters related to the initrd llike real_root=, root=/dev/ram0, or init= as it will break things. just a simple ( root=/dev/vgroot/root dolvm ) should work.

[Vieri]

OK, thanks, everyone.

I do believe however that there seems to be a bug in the way the init script generated by genkernel actually initializes the volume groups, or maybe it's a kernel "thing". I don't know, but here's what I've observed.

I've booted the kernel from the grub entry that grub-install makes out of 10_linux. All boots fine untl the initramdisk tries to mount the root partition (LVM). It complains that /dev/mapper/vgroot-root does not exist.
Now, if I enter the SHELL manually, I can see that /dev/mapper/ is empty (except for "control") and /dev/vgroot is non-existent. I also see from dmesg that before trying to mount /dev/mapper/vgroot-root the boot process actually did try to initialize LVM volume groups. I see the following:

Code: Select all

Executed: 'lvm vgscan'
Executed: 'udevadm settle --timeout=120'

and then:

Code: Select all

Executed: 'lvm vgchange -ay --sysinit'
Executed: 'udevadm settle --timeout=120'
Executed: 'mkdir -p /newroot'

Finally:

Code: Select all

[OK] Determining root device (trying /dev/mapper/vgroot-root) ...
[!!] Block device /dev/mapper/vgroot-root is not a valid root device
[!!] Could not find the root block device in /dev/mapper/vgroot-root

However, within the "rescueshell" I can manually run:

Code: Select all

# lvm vgchange -ay --sysinit
  1 logical volume(s) in volume group 'vgroot' now active

Now, both /dev/mapper/ and /dev/vgroot/ contain the devices I'm expecting.

Simply exiting the "rescueshell"resumes the boot process, and my root partition mounts fine bringing me to a login prompt.

So, I guess something screws up at boot time. It might be a timeout, the boot process is too fast, I don't know...
If no one has seen this before I guess I'll have to open a bug report.

Thanks

[skellr]

Maybe rootdelay would help with that.

Code: Select all

rootdelay[=<...>], rootwait[=<...>]
           Pauses for up to 3 seconds (or specified number of seconds) while waiting for root
           device to appear during initramfs root scanning.

Good luck.

[Vieri]

Thanks for the suggestion, but I don't think this will make any difference at all (I'll try it as soon as I get the chance though).

The reason I'm saying this is that the init script that Genkernel generates already does a while loop to wait for the root device, and it seems to me that aprox. 3 seconds should be more than enough.
In any case, even if the while loop breaks and I enter the "rescueshell" manually, a whole lot of time has already passed until I manually inspect /dev/mapper and before I run vgchange -ay again... So no matter how big the delay, the root partition will never show up unless vgchange is run again.

So I'm wondering why the first run within Genkernel's init script failed to bring ROOT up.
Unfortunately, I didn't take a peak at /run/initramfs/init.log when I was in the "rescueshell", and I won't be able to do so for the next two days.
Instead, I ran vgchange -ay manually, exited the shell and finished booting the system as expected.

Now, does anyone know if it's possible to access init.log from the booted system, or has the initramdisk been wiped out of memory?

[EDIT] Maybe the init.log can tell me what went wrong when the init script tried to scan and init the volumes. Also, I might try to use the scandelay= kernel line option.

[NeddySeagoon]

Vieri,

The initramfs is abandoned when it exits with the pivotroot command.
The may be a kernel command line parameter to keep it.

[alamahant]

Is your /boot a separate partition?
If it is on encrypted lvm then you need to add additional parameters to /etc/default/grub
It is not a good idea to have encrypted or lvm /boot
Maybe add

Code: Select all

GRUB_ENABLE_CRYPTODISK=y
GRUB_PRELOAD_MODULES="part_msdos part_gpt luks luks2 lvm"
 

Is your /dev/vgroot/root encrypted?
Did you use luks1 or luks2?
Your /boot should ONLY be luks1
Can you please post the output of

Code: Select all

lsblk -fs

[Vieri]

I thought initramfs was gone too. So I guess I'll need to reboot and check init.log to see if I can get something out of it. I will also try to add the scandelay= option, but that will be in a few days.

Meanwhile, the output of blkid (run after running vgchange manually, exiting "rescueshell" and logging in the system) is:

Code: Select all

# lsblk -fs
NAME                 FSTYPE            FSVER LABEL    UUID                                   FSAVAIL FSUSE% MOUNTPOINT
sda2                 vfat                             A483-1682
└─sda
sdb2                 vfat                             A485-1FD7
└─sdb
sdc1                 vfat                    KINGSTON 3C61-F846
└─sdc
md1
├─sda1               linux_raid_member                50191324-5a36-9401-cb20-1669f728008a
│ └─sda
└─sdb1               linux_raid_member                50191324-5a36-9401-cb20-1669f728008a
  └─sdb
md3                  ext2                             e54bbc62-bd17-487a-853a-ed14d0702a71
├─sda3               linux_raid_member                a1c952fe-ccf5-ea8d-cb20-1669f728008a
│ └─sda
└─sdb3               linux_raid_member                a1c952fe-ccf5-ea8d-cb20-1669f728008a
  └─sdb
md4                  swap                             cdb2cd63-f80d-4dc0-b71f-32040e820de2                  [SWAP]
├─sda4               linux_raid_member                644f2899-7ab5-22bd-cb20-1669f728008a
│ └─sda
└─sdb4               linux_raid_member                644f2899-7ab5-22bd-cb20-1669f728008a
  └─sdb
vgroot-root          ext4                             f6ad07cc-7a5c-4b62-9783-15f36e9ac00a    158.6G     8% /
├─vgroot-root_rmeta_0
│
│ └─sda5             LVM2_member                      PySdiK-NJWq-xiDr-vnxV-Eam7-eJ8f-gsGXHn
│   └─sda
├─vgroot-root_rimage_0
│                    ext4                             f6ad07cc-7a5c-4b62-9783-15f36e9ac00a
│ └─sda5             LVM2_member                      PySdiK-NJWq-xiDr-vnxV-Eam7-eJ8f-gsGXHn
│   └─sda
├─vgroot-root_rmeta_1
│
│ └─sdb5             LVM2_member                      xFq6vW-lECT-2VUY-ZqNJ-9tIZ-uC55-dlFbnS
│   └─sdb
└─vgroot-root_rimage_1
                     ext4                             f6ad07cc-7a5c-4b62-9783-15f36e9ac00a
  └─sdb5             LVM2_member                      xFq6vW-lECT-2VUY-ZqNJ-9tIZ-uC55-dlFbnS
    └─sdb

So, yes, /boot is a seperate partition and is in mdadm RAID1.
The ROOT LVM is not encrypted.
No LUKS.

[alamahant]

Would it be awfully terrible if you created a simple plain no-raid /boot partition and used that instead of the raid one?
Because as I see it you are now struggling with two tricky things instead of one.
/ on raid lvm
and /boot on raid.
Just eliminate the second and see if the first also is rectified.
You will need ofcourse to reinstall and update grub and create a new initramfs.
Maybe you can try entering

Code: Select all

GRUB_PRELOAD_MODULES="part_msdos part_gpt mdraid1x lvm"

in /etc/default/grub
before changing your partitions though....

[Vieri]

The kernel parameter scandelay solved the issue.

If I install grub with the following then all works fine:

Code: Select all

# grep ^GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="domdadm dolvm scandelay=1"

Without scandelay=1 (or more) initramfs is unable to start the volume groups.