SOLVED! Kernel Panic on 6.9.x Upgrade

Message

Vorlon · Post by **Vorlon** » Sun May 19, 2024 6:33 pm

I'm running a monolithic kernel on ~amd64, as I have done for about a decade. I routinely upgrade the gentoo-sources kernel as they become available. Recently, I've been getting a kernel panic for 6.9.0 and 6.9.1. Kernel 6.8.9 runs fine.

Reviewing the dmesg output for the 6.9.x kernels vs correctly working 6.8.9, I see that the new kernel doesn't seem to try to access my nvme disk, but looks at the SATA disks instead. It eventually gives me a "/dev/root: Can't open blockdev" error and then panics.

I've booted the Gentoo live disk image and verified that all the kernel modules loaded by the live system are selected in my .config file. I've also used the hwinfo utility to list all the required modules, and have verified that they are selected for inclusion in the kernel.

Something has changed with the 6.9.x kernel, but I can't figure out what it is. I've got all the nvme options selected to include in my monolithic kernel.

Can somebody suggest something else to try?

TIA!

Post by **NeddySeagoon** » Sun May 19, 2024 6:57 pm

Vorlon,

Post your

Code: Select all

lspci -nnk

so we can see your hardware and put your broken .config onto a pastebin site please.

Vorlon · Post by **Vorlon** » Sun May 19, 2024 7:52 pm

Thank you for your help!!

lspci -nnk

Code: Select all

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
	Subsystem: ASRock Incorporation Device [1849:1450]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Kernel driver in use: pcieport
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
	Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
	Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
	Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
	Subsystem: ASRock Incorporation Device [1849:790b]
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
	Subsystem: ASRock Incorporation Device [1849:790e]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
	Kernel driver in use: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
01:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
	Subsystem: Intel Corporation Device [8086:390d]
	Kernel driver in use: nvme
02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43d0]
	Kernel driver in use: xhci_hcd
02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c8]
	Kernel driver in use: ahci
02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c6]
	Kernel driver in use: pcieport
03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c7]
	Kernel driver in use: pcieport
03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c7]
	Kernel driver in use: pcieport
03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c7]
	Kernel driver in use: pcieport
03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c7]
	Kernel driver in use: pcieport
03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASRock Incorporation Device [1849:43c7]
	Kernel driver in use: pcieport
03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:3306]
	Kernel driver in use: pcieport
03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:3306]
	Kernel driver in use: pcieport
05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
	Subsystem: ASRock Incorporation Device [1849:1539]
	Kernel driver in use: igb
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	Subsystem: Tul Corporation / PowerColor Radeon RX 570 4G [Red Dragon] [148c:2379]
	Kernel driver in use: amdgpu
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	Subsystem: Tul Corporation / PowerColor Device [148c:aaf0]
	Kernel driver in use: snd_hda_intel
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
0c:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device [1022:1456]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device [1022:1456]
0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 xHCI Compliant Host Controller [1022:145f]
	Subsystem: ASRock Incorporation Device [1849:7914]
	Kernel driver in use: xhci_hcd
0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
0d:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
	Subsystem: ASRock Incorporation Device [1849:7901]
	Kernel driver in use: ahci
0d:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
	Subsystem: ASRock Incorporation Device [1849:2220]
	Kernel driver in use: snd_hda_intel

.config is here: https://pastebin.com/vjz08YRx

Swapped size=9 tags to code tags for easy reading -- NeddySeagoon

New at Gentoo · Post by **pietinger** » Sun May 19, 2024 9:24 pm

Maybe it is not your new kernel ... maybe it is your bootmanager/bootloader ... giving wrong kernel command line parameters to your kernel ...

Do you use grub ? If yes, I would like to see the /boot/grub/grub.cfg AND the output of "blkid" (and the information which of your partitions is your root partition).

Vorlon · Post by **Vorlon** » Sun May 19, 2024 9:39 pm

Hmmm... interesting idea. Grub has been kinda goofy lately. (I've had to mask some version for my other stable gentoo systems.)

Grub.cfg is here: https://pastebin.com/973DdzWY

Here is blkid

Code: Select all

/dev/nvme0n1p3: LABEL="Landru" UUID="b3f7e921-d3c1-4e04-a614-6450158a19ff" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="cd217cee-2d83-d044-91b6-a875d504eaa4"
/dev/nvme0n1p1: LABEL_FATBOOT="UEFI" LABEL="UEFI" UUID="3C9B-E0C2" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="b5b8a2f8-4a6c-1f48-b892-972fe206ed2d"
/dev/nvme0n1p2: UUID="5972ebe8-9667-4500-bfae-d2fcaaeba2be" TYPE="swap" PARTUUID="38f95a2f-6d5b-2547-ba32-8bc5b9ec7b95"
/dev/sdc1: LABEL="temp" UUID="3ed139a8-4b2b-4b03-bad1-c047435eff64" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="f75e1f4c-01"

Thank you for your help!!

New at Gentoo · Post by **pietinger** » Sun May 19, 2024 10:19 pm

Hmmm ... grub.cfg looks good ... yes, I see you have every module which kernel needs to access its root partition in your .config (EFI_PARTITION, BLK_DEV_*, SCCI, NVME, AHCI, FS) ... but you have activated some options that I don't know if they cause problems (they are options that are hardly ever used) ...

Do you really use CONFIG_POWER_RESET and CONFIG_NVMEM_REBOOT_MODE=y ?

And these: CONFIG_NVME_KEYRING=y and CONFIG_NVME_AUTH=y ?

You said you are using a monolithic kernel (like me) ... but do use also an initramfs ?

Because you have enabled it (CONFIG_BLK_DEV_INITRD=y) AND grub gives also an initramfs to your kernel (/boot/amd-uc.img)

When a kernel has an initramfs THEN it is the job of this initramfs to mount the rootpartition ... but you said you had a kernel panic ...

I would need additionally the "dmesg" (after booting one of your old kernels) to see more.

BTW: I have seen this:

Code: Select all

1.
CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL=y
2.
# CONFIG_PINCTRL_AMD is not set
3.
# CONFIG_AMD_PTDMA is not set
4.
# CONFIG_IRQ_REMAP is not set

1. Please read the 2nd part of this chapter:
https://wiki.gentoo.org/wiki/User:Pieti ... Must_Haves
2. Only important if you have devices which use i2c
3. Maybe check if you use it (performance)
4. THIS CAN BE IMPORTANT ... usually it should be enabled always ... but maybe also not the reason of your problem.

P.S.: Have you checked how much space you have left on your boot partition (you have many old kernels) ?

Post by **NeddySeagoon** » Mon May 20, 2024 10:47 am

Vorlon,

You have 20 polaris10 firmware files listed in your kernel and

Code: Select all

ls /lib/firmware/amdgpu/polaris10* | wc -l
21

so AMD have added one.

It will probably go if you provide all the firmware.
I've been there and done that too :)

Vorlon · Post by **Vorlon** » Mon May 20, 2024 10:24 pm

Update 2024-05-20:

Things tried:
1. Ensured I have all the "Must Haves" in pietinger's write up. (Very nice write-up, BTW!)
2. Selected CONFIG_IRQ_REMAP = Y
3. De-Selected CONFIG_BLK_DEV_INITRD. (Kernel notes say "mark Y if unsure")
4. Added missing polaris firmware file to kernel. (Good work spotting a very subtle error, NeddySeagoon!)
5. Ensured all the selected items in NVMe wiki are set (https://wiki.gentoo.org/wiki/NVMe)
6. Verified sufficient space in /boot (/boot is part of the main partition). Also ensured /efi has free space.
7. Rebuilt gentoo-sources 6.8.9, so it's (probably) not a grub issue

Results: Still the same kernel panic.
I can still boot to kernel 6.8.9, but 6.9.0 & 6.9.1 cause panic. <sigh>

So I'm still running, but perplexed.

Much appreciate all the suggestions!

Post by **NeddySeagoon** » Tue May 21, 2024 10:17 am

Vorlon,

Please pastebin the changed kernel .config file and the kernel panic reason.
The message will say Kernel Panic ...

New at Gentoo · Post by **pietinger** » Tue May 21, 2024 4:14 pm

Vorlon,

the reason why I wanted to check the grub first is as follows:

I have seen that you have all the modules necessary for the kernel to access the root partition. And there are actually only two ways to cause a kernel panic during a CONFIGURATION: One of these modules is missing (or you configure the kernel for the wrong CPU architecture). In addition, it should not be possible to create an incorrect configuration if you already have an (old) working one and then only use this with "make oldconfig". If kernel configuration AND grub are correct, there is of course still the (very small) chance that we have a regression in the kernel. But before we assume this and create a kernel bug report, I wanted to have a look at the "dmesg" of the previous (working) kernel, because I see a lot about the system here:
https://wiki.gentoo.org/wiki/User:Pieti ... tion#dmesg

(I am also already on 6.9.1 and have no problems at all. Maybe you want to see what's new:
https://wiki.gentoo.org/wiki/User:Pieti ... ersion_6.9 )

Vorlon · Post by **Vorlon** » Tue May 21, 2024 5:48 pm

pietinger,

Here is my dmesg for a good boot using Kernel 6.8.9. https://pastebin.com/iaPjzJx7
A screen shot of the boot process for kernel 6.9.1 is here: https://pasteboard.co/cwW5BQD4aaus.jpg

NeddySeagoon,

Here is the .config file for 6.9.1:https://pastebin.com/YZuKwLHL

Thank you both for your continued help!

keltar · Post by **keltar** » Tue May 21, 2024 6:16 pm

I have similar problem on my laptop but not on desktop system, running nearly identical systems, with the same .config. Not sure if it is the same problem though. I'll look into it when I have time. In my case the error code for root mount is -16, which seems to by EBUSY.

Goverp · Post by **Goverp** » Wed May 22, 2024 8:07 am

This is probably irrelevant, but I started using the 6.9 series when 6.9.1 went stable. First boot seemed fine, but the second boot hung. After some investigation, it turned out it wasn't hanging, it was heavily delayed by my initramfs's "fsck" of my rootfs. Under previous kernels that was a short run that took a second or two to be happy. On 6.9.1, after the first boot, it now spots some error (I forget what) in the metadata, which triggers a full "fsck -f" that takes a minute or two.

Note that this was on an f2fs root partition, and is probably specific to that. I've not had time to investigate further - not too keen on bisecting something that's presumably slightly corrupting my rootfs!

Post by **NeddySeagoon** » Wed May 22, 2024 4:34 pm

Vorlon,

Your image shows the kernel trying to mount 103:3 as root.
That's the major:minor. device number.

/usr/src/linux/Documentation/admin-guide/devices.txt says that is

Code: Select all

 103 block      Audit device
                  0 = /dev/audit        Audit device

which is not NVMe.

My kernel NVME setup is

Code: Select all

$ grep NVME /usr/src/linux/.config
# NVME Support
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
# CONFIG_NVME_MULTIPATH is not set
# CONFIG_NVME_VERBOSE_ERRORS is not set
CONFIG_NVME_HWMON=y
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
# CONFIG_NVME_HOST_AUTH is not set
# CONFIG_NVME_TARGET is not set
# end of NVME Support
CONFIG_RTC_NVMEM=y
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y
# CONFIG_NVMEM_LAYOUTS is not set
# CONFIG_NVMEM_RMEM is not set

grknight · Post by **grknight** » Wed May 22, 2024 5:33 pm

The photo shows a line Can't open blockdev which comes from file system mounting.
This changed in 6.9 with this commit as its start
For whatever reason, the symbolic /dev/root isn't pointing to the NVMe partition. This should be updated into the DEVTMPFS.

https://github.com/torvalds/linux/blob/ ... dev.c#L986 is now the function responsible for opening devices.

Since certain CGroup options are not set, the ret = bdev_permission(dev, mode, holder); always returns true.

I would suggest adding some pr_warn("<message>") macros into the bdev source above before each step to know where it stops. I only say this because there isn't much talking out via printk (what dmesg shows) even with a debug flag that I can tell.

The other alternative is to use an initramfs, like dracut, to control the mounting and would also give a rescue shell to inspect /dev.

keltar · Post by **keltar** » Wed May 22, 2024 6:24 pm

Thanks for the hint grknight.
In my case it failed at bdev_open. But interesting thing is it was called twice, and failed on second call. So I digged a bit and managed to boot 6.9.1 after removing resume= and resume_offset= options. I've also stumbled upon https://bugzilla.kernel.org/show_bug.cgi?id=218845 which may be about the same problem I had.

But Vorlon's grub config don't seem to have resume option, so it is probably different problem from mine.
Do you have an error code specified somewhere near error message? Image seems to be clipped so can't tell.

New at Gentoo · Post by **pietinger** » Wed May 22, 2024 6:55 pm

keltar wrote:Thanks for the hint grknight.

+ 1 !

keltar wrote:But Vorlon's grub config don't seem to have resume option, so it is probably different problem from mine.

Yes ... but maybe it is related ... he has enabled CONFIG_POWER_RESET (which is usually only senseful for risc (arm) machines) ... I never had this option enabled for an Intel or AMD machine.

Vorlon · Post by **Vorlon** » Wed May 22, 2024 10:41 pm

Woo Hoo! I'm booting to 6.9.1.

I tried removing the CONFIG_POWER_RESET, but it didn't make a difference.

The fix was to disable suspend and resume functions in the kernel. (I usually don't use that anyway.)

Much thanks to all who dug into this and offered suggestions!!

New at Gentoo · Post by **pietinger** » Wed May 22, 2024 10:46 pm

Vorlon wrote:The fix was to disable suspend and resume functions in the kernel. (I usually don't use that anyway.)

Thank you also for your report !

Have you also tried the patch from the bug report ? ( https://bugzilla.kernel.org/attachment. ... ction=diff )

For me this is a clear regression ... maybe it will be already fixed in 6.9.2 ... I will read the patches for 6.9.2 very carefully (as soon as available).

keltar · Post by **keltar** » Thu May 23, 2024 2:15 pm

pietinger, linked patch works fine for me

New at Gentoo · Post by **pietinger** » Thu May 23, 2024 3:13 pm

keltar wrote:pietinger, linked patch works fine for me

Yes, great.

Our developer @Grknight has also updated our wiki page in the meantime:
https://wiki.gentoo.org/index.php?title ... id=1285831

Vorlon · Post by **Vorlon** » Sun May 26, 2024 10:44 am

FYI. Kernel 6.9.3 worked for me, even with the hibernation and sleep enabled in the kernel.

grknight · Post by **grknight** » Sun May 26, 2024 1:12 pm

Vorlon wrote:FYI. Kernel 6.9.3 worked for me, even with the hibernation and sleep enabled in the kernel.

It will break if there is any error reading/applying the suspend-to-disk information. 6.9.3 does not yet exist but .2 does not yet have the fix. The next release may.

Vorlon · Post by **Vorlon** » Sun May 26, 2024 1:16 pm

Ooops. You're right. For some reason portage asked for the kernel to be updated. I assumed this was the next version bump.

Sorry.