[Solved] Kernel Panic, Not syncing VFS, unable to mount root

[Knov]

Hello, I had installed a full disk encryption setup using luks and lvm and had stayed on the same kernel for a long time before deciding to update, going through all the typical motions. However, I think I must have forgotten an important step related to luks and lvm when regenerating the initramfs or grub, as it was my first time upgrading my kernel on a system that was fully encrypted. When I upgraded my kernel I used my old configuration. Now I have a kernel panic and my system will not boot.

The bootloader information is as follows

Code: Select all

/dev/root: can't open blockdev
VFS: Cannot open root device "mapper/vg0-root" or unknown-block(0,0): error -6
Please append a correct "root=" boot option

It also says something about a kernel offset.
I've already tried to chroot in and rebuild my kernel, regenerate initiramfs with lvm and luks options, I've checked my grub configurations and I have the necessary configurations related to fde in it, and I've updated grub. I don't know what to do. Any help much appreciated,
thank you
(also I will promptly post kernel configs etc when requested I just don't know what is relevant right now).

[NeddySeagoon]

Knov,

That you get a kernel panic at all tells that the initrd is not being used. When things go wrong, the initrd provides a rescue shell, so that you can poke about and see why

Code: Select all

unknown-block(0,0)

tells that the kernel cannot see any block devices at all. That's expected using luks and lvm as the kernel modules to read block devices are in the initrd.

How did you do the kernel update?
How did you build the matching initrd?
The two are tied together by kernel module version magic.

How did you update grub.cfg?

[Knov]

Thank you for the reply. I just chrooted into the system and did as follows

How did you do the kernel update?

First I properly chrooted into the system, decrypted my root partition, mounted all relevant drives and psuedo filesystems following the gentoo handbook.
I then ran these commands

Code: Select all

emerge --sync
emerge -avuDN @world
emerge --depclean

eselect kernel list
eselect kernel set 1 (this was linux-6.5.1)
cd /usr/src/linux

I then copied over my kernel .config backup into the directory for linux-6.5.1

Code: Select all

make olddefconfig
make && make modules_install && make install
emerge @module-rebuild

How did you build the matching initrd?

Code: Select all

genkernel --lvm --luks initramfs

When I generated my initramfs this was outputed.

Code: Select all

current kernel's LOCALVERSION is set to ''; Will ignore set --kernel-localversion value '-x86_64' because kernel was not built
...

Which sounds quite ominous
Then finally

Code: Select all

grub-mkconfig -o /boot/grub/grub.cfg

Which produced no errors. I then exited and dismounted. Thank you for your reply again.

[NeddySeagoon]

Knov,

I'm with you so far ...

What do you have in /boot/grub/grub.cfg. That's a pastebin please.
What does

Code: Select all

ls -l /boot

have to say?

It looks like you have all the bits but that they are not being put together.

[Knov]

Here is my grub.cfg at /boot/grub/grub.cfg
grub.cfg
And

Code: Select all

ls -l /boot

outputs this
ls -l /boot
Thank you

[NeddySeagoon]

Knov,

I was expecting to see grub.cfg missing the initrd, or either the new kernel or initrd files truncated.
Its none of that.

I've heard some developer chatter on IRC lately that suggests that genkernel is on life support and that dracut should be used for building the initrd.
I've never used either.

Try dracut to build your initrd.

It looks like the initrd you have is broken. grub.cfg loads it.

I'm out of ideas really. Dracut is about all I have left.

[Knov]

Try dracut to build your initrd.

Will do, thank you for your help.
This is what I just attempted following the section regarding dracut in this guide on the Gentoo wiki
I added this to /etc/dracut.conf

Code: Select all

add_dracutmodules+=" crypt dm rootfs-block "

I then ran

Code: Select all

dracut --force --kver 6.5.1-gentoo

I then remade grub.
Still a kernel panic, I'm not exactly sure how to use dracut, what did I miss?

[NeddySeagoon]

Knov,

You have used dracut more than me.
I make my initrd like the DIY Wiki Page
That's not a recommendation.

What did dracut call your new initrd?
Is it in /boot ?
Is grub.cfg pointing to it?

Its interesting that you still have a kernel panic. That makes me think that your new initrd is not in use.

/boot, pre dracut, listed

Code: Select all

-rwxr-xr-x 1 root root 15290160 May 23 23:19 initramfs-6.1.28-gentoo.img
-rwxr-xr-x 1 root root 13735838 May 23 23:17 initramfs-6.1.28-gentoo.img.old
-rwxr-xr-x 1 root root  5376172 Sep  3 15:17 initramfs-6.4.11-gentoo.img
-rwxr-xr-x 1 root root  5376848 Sep  3 15:16 initramfs-6.4.11-gentoo.img.old
-rwxr-xr-x 1 root root  5350396 Sep  4 00:10 initramfs-6.5.1-gentoo.img

and grub.cfg used initramfs-6.5.1-gentoo.img[ for your new kernel.

[sublogic]

... First I properly chrooted into the system, decrypted my root partition, mounted all relevant drives and psuedo filesystems following the gentoo handbook.

This doesn't solve your problem, but rather than chrooting from rescue media, you could boot one of your older kernels. In fact, why don't you do that, just to make sure they still work. In the grub menu, you should be able to scroll to "Advanced options", press ENTER and pick one of your older kernels. (Always keep a known good kernel !)

Like NeddySeagoon, I am at a loss. Your 6.5.1 kernel is booting, but your 6.5.1 initramfs isn't used. Yet it is present in /boot and is listed in grub.cfg .

In your grub.cfg:

linux /vmlinuz-6.5.1-gentoo root=/dev/mapper/vg0-root ro nvidia-drm.modeset=1 dolvm rootdelay=3 crypt_root=PARTUUID=37799cf5-2c9e-4ab7-9a53-5802e12fa495 root_trim=yes rd.luks.options=discard

Nitpick: rd.luks.options is a dracut-ism and you are (still?) using genkernel. If I'm right the genkernel equivalent is crypt_root_options=--allow-discards --but that is not your immediate problem.

[Knov]

In the grub menu, you should be able to scroll to "Advanced options", press ENTER and pick one of your older kernels. (Always keep a known good kernel !)

Wow didn't realise I could do that, and yes I was able to boot into my old kernel 6.1.28 (which was the kernel I was using until I upgraded and the kernel panic happened). In the interim, I tried downgrading to lower known stable version linux-6.1.46-gentoo but it again resulted in a kernel panic. So 6.1.28.old fails, 6.1.28 works and all the rest panic badly.

What did dracut call your new initrd?
Is it in /boot ?
Is grub.cfg pointing to it?

Well like I said just above in response to sublogic, even though I downgraded to 6.1.46, when I booted my computer it booted me into 6.5.1. Sorry I did too much stuff to properly respond to you. Anyway 6.1.28 works, and Xserver is working. But what can we do with that information?

edit: As of right now I've removed all non working related kernel versions and I'm successfully booting into 6.1.28, which works. That version of the kernel is masked and vulnerable to stack rot apparently. But I don't know where to go from here, I obviously want to be able to upgrade my kernel, and I will be switching to an amd gpu soon.
Thank you for your help.

[NeddySeagoon]

Knov,

Build your 6.5.x kernel.
Use dracut to build a matching initrd.
Don't update grub.cfg yet

With all the bits, installed, what does

Code: Select all

ls -l /boot

show?

I suspect dracut initrd naming is not what you or the autoblackmagic expect.

If the kernel and initrd are in /boot update grub.cfg and post it.

I'm half expecting the boot stanza for the 6.5.x kernel to be missing the initrd entry entirely.

dracut fails to generate initrd may be worth following too.

[Knov]

Knov,
With all the bits, installed, what does

Code: Select all

ls -l /boot

show?

ls -l /boot

If the kernel and initrd are in /boot update grub.cfg and post it.

They were both in boot so here is grub.cfg
It kernel panicked again. Could it maybe be that the config file I'm using to build my kernels is corrupted in some way?

[Knov]

Knov,
With all the bits, installed, what does

Code: Select all

ls -l /boot

show?

ls -l /boot

If the kernel and initrd are in /boot update grub.cfg and post it.

They were both in boot so here is grub.cfg
It kernel panicked again. Could it maybe be that the config file I'm using to build my kernels is corrupted in some way? Or maybe I should upload screenshots of the boot message?

[sublogic]

If the kernel and initrd are in /boot update grub.cfg and post it.

They were both in boot so here is grub.cfg
It kernel panicked again. Could it maybe be that the config file I'm using to build my kernels is corrupted in some way? Or maybe I should upload screenshots of the boot message?

Running out of ideas, but one thing we haven't checked: was /boot mounted when you installed the kernel and updated the grub.cfg ?

That is, are the files in the /boot partition, or are they in the /boot directory of the root partition ?

Code: Select all

# mount | grep boot
# ls -l /boot
# mount /boot
# ls -l /boot

If the two "ls -l" output are different,

Code: Select all

# cd /usr/src/linux
# make install                           ;: while /boot is still mounted !
# dracut --force --kver 6.5.1-gentoo     ;: or whatever the right dracut command is
# grub-mkconfig -o /boot/grub/grub.cfg

and then, to clean up the false install in the root partition,

Code: Select all

# umount /boot
# cd /boot
# rm -rf *

(Looks scary, doesn't it? but this is to leave an empty /boot directory in the root partition. Good practice. When you mount /boot, its content hides whatever was in the directory so leaving anything there wastes space.)

And if /boot was mounted all along, ignore the above. Just another false lead.

[Knov]

And if /boot was mounted all along, ignore the above. Just another false lead.

Unfortunately it was mounted properly all along.

Code: Select all

ls -l /boot

when I umount the partition associated with boot shows an empty directory.

[sMueggli]

The sizes of the initramfs are quite different:

Code: Select all

-rwxr-xr-x 1 root root 15290160 May 23 13:19 initramfs-6.1.28-gentoo.img
-rwxr-xr-x 1 root root 13735838 May 23 13:17 initramfs-6.1.28-gentoo.img.old
-rwxr-xr-x 1 root root  5347640 Sep  5 08:18 initramfs-6.5.1-gentoo.img

Are you sure that you have enough free space on your ESP?

What is the size of a initramfs created by genkernel with the luks and lvm options?

[Knov]

Are you sure that you have enough free space on your ESP?

Yes, if this means anything.

Code: Select all

/dev/nvme0n1p1        511M  163M  349M  32% /boot

What is the size of a initramfs created by genkernel with the luks and lvm options?

Code: Select all

-rwxr-xr-x  1 root root  15M May 23 13:19 initramfs-6.1.28-gentoo.img
-rwxr-xr-x  1 root root  14M May 23 13:17 initramfs-6.1.28-gentoo.img.old
-rwxr-xr-x  1 root root 5.2M Sep  5 15:56 initramfs-6.5.1-gentoo.img
-rwxr-xr-x  1 root root  32M Sep  5 08:54 initramfs-6.5.1-gentoo.img.old

Seems to be a lot smaller.

[Knov]

My sys-boot/grub was installed without the USE flags "device-mapper" or "mount", do you think this could be potentially causing the problem? I don't know how it was working in the first place if it was necessary, but the guide I was following made sure to set these use flags before emerging and installing grub. I just don't want to do this unless it could potentially be the cause of the kernel panic problem as I at least can get into 6.1.28 at the moment, and don't want to mess that up. I'll run this if you think it is a solution.

Code: Select all

grub-install --target=x86_64-efi --efi-directory=/boot 
genkernel --lvm --luks initramfs
grub-mkconfig -o /boot/grub/grub.cfg

Or use dracut instead.

[NeddySeagoon]

Knov,

Code: Select all

		echo	'Loading Linux 6.5.1-gentoo ...'
		linux	/vmlinuz-6.5.1-gentoo root=/dev/mapper/vg0-root ro single nvidia-drm.modeset=1
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-6.5.1-gentoo.img

The echo statements print a massage about what grub in about to do.

The linux and initrd statements do it.
Thats all correct.

If you still get a kernel panic, your initrd is loaded but not used.

Does the kernel support the use of an initrd and is the initrd compressed is a way that the kernel understands?

[Knov]

Thats all correct.

Okay.

If you still get a kernel panic, your initrd is loaded but not used.
Does the kernel support the use of an initrd and is the initrd compressed is a way that the kernel understands?

I honestly don't know sorry, here is my .config. Thank you for continuing to help. If you cannot spot anything wrong, maybe I should use a default genkernel kernel and see if that works?

[NeddySeagoon]

Knov,

I don't know how to give up :)

Code: Select all

CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_RD_GZIP is not set
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_XZ is not set
# CONFIG_RD_LZO is not set
CONFIG_RD_LZ4=y
# CONFIG_RD_ZSTD is not set
# CONFIG_BOOT_CONFIG is not set
CONFIG_INITRAMFS_PRESERVE_MTIME=y

That's initrd support but only lz4 compression.
If your initrd uses one of the other compression algorithms, the kernel will not read it.
What does file say about your initrd?

Code: Select all

moriarty ~ # file /boot/initramfs 
/boot/initramfs: ASCII cpio archive (SVR4 with no CRC)

I think CPIO support is free. Mine works but I have all the kernel decompression support on.

[Knov]

I don't know how to give up

I think you just found something super promising,

Code: Select all

knov /boot # file initramfs-6.*
initramfs-6.1.28-gentoo.img:     LZ4 compressed data (v0.1-v0.9)
initramfs-6.1.28-gentoo.img.old: LZ4 compressed data (v0.1-v0.9)
initramfs-6.5.1-gentoo.img:      XZ compressed data, checksum NONE
initramfs-6.5.1-gentoo.img.old:  ASCII cpio archive (SVR4 with no CRC)

I'm pretty sure that the non .old 6.5.1 is a genkernel initrd, and the .old was a dracut generated one. I'm guessing I build the other decompression support into my 6.5.1 kernel? Or is there a way to generate the initrd as LZ4 compressed data specifically?

[NeddySeagoon]

Knov,

You may be able to tell dracut what compressor to use.
You can probably decompress/recompress the initrd too but thats faffing about every time.

Just fix your kernel so it won't matter.

[Knov]

Added the support for the compression algorithms and successfully booted into 6.5.1. Thank you everyone for helping me and thank you for figuring out the issue NeddySeagoon. Thanks again.

[NeddySeagoon]

Knov,

That's experience for you.
- it's what you get just after you needed it. :)

On to your next Gentoo learning opportunity.