Gentoo install ISO boot with 'Kernel Panic' [ROOT CAUSED]

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Hi all !

When I boot minimal ISO (install-amd64-minimal-20240421T170413Z.iso) builded on top of 6.6.21-gentoo-x86_64 kernel it fails with Kernel Panic just after udev activation (sometimes later). Here are screenshots:

https://ibb.co/zhpWQZg
https://ibb.co/D5Vy11t

My config is:
Core i7 12700K with MSI Z690 DDR4 Motherboard, 32GB of DDR4 3200 RAM, 2xNVMe Lexar discs, 1xSATA disc.

I tried all available minimal and admin ISO images. All of them mostly fail with that Kernel Panic. Sometimes just hang during boot and sometimes they can even coot and everything seems to be fine. I tried with nosmp, also tried to boot on only one core (disabled all but one in BIOS) but without effect.

I also tried Funtoo installation ISO and it boots everytime (it uses linux kernel 5.X.X). Windows 11 Professional I have also works correctly. I just wanted to update my Gentoo installation from scrath. Previously I installed from ISO builded on the top of kernel 5.5.x and my old installation worked really fine. Unfortunately I deleted old ISO - my bad. Is there any archive of old Gentoo install ISO ?

The second question - does anybody have older minimal ISO image ? I remember that when I installed Gentoo on that system with kernel 5.5.X it was really stable.

Thanks in advance :-)

_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Sat Apr 27, 2024 12:07 pm Post subject:

Sławomir Gąsiorowski,

I can host any or all of

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Thank you for response. I tried some kernel options, but none of them helped (irq-pool, noapic, nolapic, nohotplug, nosata, nosmp ...). Installer sporadically boots fine, but mostly it fails. I tried Debian Live DVD 12.5 that was builded on top of the linux kernel 6.1.x and it runs great. I don't think it's a hardware problem. I'm going to use Debian Live USB to install Gentoo and I will try with different kernels versions and config. I will put my feedback here.
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Sat Apr 27, 2024 7:00 pm Post subject:

Sławomir Gąsiorowski,

That sounds like a plan.
You can probably put the Debian kernel under your Gentoo install as a get-U-going measure too.

That's the kernel, initrd and modules, not just the kernel.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Hi, have very interesting feedback. First of all It turned out that Debian Live USB I use was build on top of kernel 6.x and I was able to reproduce exactly the same Kernel error and it is related with NVMe driver. Problem starts reproducing when I start working on partition, just after mounting it. And no matter what filesystem I used. After all I was able to reproduce it very often always just after nvme partition mount or later during massive I/O activity on nvme partition.

The tool smartctl shows no errors and disk seems to be in good condition. Additionaly I performed extended SMART test and also no errors found. I decided to test it under Windows 11. I created NTFS parttition and was able to fill it with data 100% then copy, delete it and fill again. No problems observed.

In my opinion linux 6.x.x has some kind of reggression in NVMe driver maybe strictly related only to model I use: Lexar NM620 512GB 2280 PCI-E 3.0. @NeddySeagoon can you share with me some Gentoo install iso from 2022/2023 that contains linux 5.x ? It's a pity that Gentoo don't host older instalation iso...
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Sun Apr 28, 2024 11:07 am Post subject:

Sławomir Gąsiorowski,

Help yourself.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

I have another finiding and probably root caused this issue. I tried some old funtoo install iso with 5.18.x kernel and again I reproduced this problem !!! So definitely reggression was introduced by something else. I bet that this is BIOS update, so I made a BIOS downgrade to some very old version from feb 2022 and it looks that it is !!! So far I don't observe problems. Before BIOS downgrade the problem was even during partition mount or stage file unpacking - hangs and kernel crash.

Thanks for the link, but first I will try to complete my installation using latest ISO installer with BIOS downgraded :-)

_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Sun Apr 28, 2024 2:16 pm Post subject:

Sławomir Gąsiorowski,

That's a good idea. That server has a lot of old stuff on it. Including all my distfiles back to mid 2006.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Hi !

I can officialy confirm. The problem with kernel crashing was caused by one of BIOS updates. I performed successfull Gentoo install using old BIOS 7D25v12 (E7D25IMS.120) from Feb 2022. So far I don't know which BIOS udpate introduced this reggression, but I will find it :-)

It looks that setup like mine was not tested by MSI enough or was never validated under Linux :evil:

This is definitely not a defect in the Linux Kernel. The defect is in the BIOS and should be fixed by MSI. I will try to report this defect to them with better evidences than one screenshot. Be careful when updating the BIOS if you have setup like mine: MSI PRO Z690-A DDR4 with Core i7 12700K and NVMe disk Lexar NM620 and want to use Linux.

Please mark this thread as [Solved]
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Mon Apr 29, 2024 10:03 am Post subject:

Sławomir Gąsiorowski,

Its the does it boot Windows?
Ship it!

school of BIOS quality control.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Hu · Administrator Joined: 06 Mar 2007 Posts: 21706

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Have more info from my testing. I narrowed the problem to my 512GB Lexar NM620 512GB M.2 2280 PCI-E x4 Gen3 NVMe. I have second one but 1TB. In theory these are the same discs but with different capacity. HwInfo shows also that there are different controllers on them and different versions of firmware. Unfortunately Lexar not provided any Firmware update for their NM620 disc series.

The failing one is: Lexar NM620 512GB M.2 2280 PCI-E x4 Gen3 NVMe (LNM620X512G-RNNNG) -> Innogrit Shasta+ IG5216 PCIe 3.0 x4 NVMe 1.4 4-channel SSD Controller (Original Device Name: Shenzhen Longsys Electronics, Device ID: 5216)
The passing one is: Lexar NM620 1TB M.2 2280 PCI-E x4 Gen3 NVMe (LNM620X001T-RNNNG) -> Shenzhen Longsys Electronics, Device ID: 1D97 (Original Device Name is the same)

So when I unplugged this 512GB disc Gentoo installer ISO boots from USB without any problem. In kernel logs I see that it detected 1TB NVMe from Lexar. When 512GB NVMe is present kernel can even immediately crash during drivers loading. I will check in linux kernel, maybe there are some "magic" knobs for that NVMe controller ?

Anyway I'm setting this thread as [ROOTCAUSED]. I'm going to update this thread when I will find any interesting info.
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

Hi all !

Have some very interesting results of my investigation. I was almost close to give up when I decided to just try how gentoo kernel binary works... So I installed one and IT WORKS !!!
I tried two stable gentoo-kernel-bin:

1. gentoo-kernel-bin-6.1.87 -> PASS
2. gentoo-kernel-bin-6.6.28 -> PASS

So I decided to just use that config from 6.6.28 binary distribution (/proc/config.gz) to reuse it on gentoo-sources-6.6.21 and it produces kernel that boots and is stable. After that I decided to make mrproper and use x86_64_defconfig from gentoo-sources-6.6.21 and after little tuning like NVMe and filesystems compilation into kernel binary it also gives me fully functional system !!!

So finally I have my Gentoo with manually compiled kernel from 6.6.21 sources fully functional with latest MSI BIOS for my MSI Z690 motherboard :-)

I also had a fruitfull discussion with my friend that actually works for Solidigm (previously for Intel like me) and he analyzed my kernel crash and pointed that it looks like some timeout in RCU (Read Copy Update feature). They are familiar with such errors. Most probably the real rootcasue is that Lexar NVMe controller or firmware not meet specifications and after BIOS update and after RCU code refactor in linux kernel some NVMe that don't meet specification may crash...

The last stage of my investigation is to find which kernel option caused that problem. Probably we are experiencing some time racing conditions in on the edge of NVMe and kernel handshake logic. When I will find something interesting I will update this thread.
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com

NeddySeagoon · Posted: Wed May 01, 2024 7:14 pm Post subject:

Sławomir Gąsiorowski,

Try a faulty kernel with

Sławomir Gąsiorowski · n00b Joined: 21 Jul 2004 Posts: 50 Location: Poland

I tried this option and as a result I got a bunch of other different errors. So this is not a solution in my case.
_________________
Slawomir Gasiorowski
email: sgasiorowski@gmail.com