how to debug early kernel hang problem

[mortonP]

I'm out of ideas.

situation:
new Arrowlake-class hardware
boots Ubuntu 25.04 from USB -> hardware is not faulty. Ubuntu comes with 6.14 kernel
does NOT boot any other way - my own kernels...

when from local NVMe storage: I get Grub menu, select kernel, and only see:

Code: Select all

EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9

...and then hangs (ctrl-alt-del does not work -> press power button)

when same kernel provided by PXE -> black screen
when booting my usual rescue kernel/system via PXE -> black screen
when booting from known-good USB SSD -> black screen
when booting either known-good 6.15.0 or 6.12.x configuration -> black screen

....this is the only system that hangs at early kernel init, all other system I have in reach boot all other variants of kernel.

This is the only system that I have that is so new - maybe Arrowlake graphics/console needs some kernel option?
(the "xe" kernel module is enabled as module)

I have already tried also booting with "nomodeset"

Googling leads to:
https://discourse.nixos.org/t/system-wo ... stub/29212

When you stop getting any output after those lines, it pretty much always means one of two things.
An extremely early kernel panic. Like super early, before it’s even managed to setup a console. Sometimes you can use the earlycon= kernel param to get some output this early. But this is very unlikely to be your problem.
The much more likely possibility is that you just need to use the correct console= kernel param. The system is probably booting as expected, but just not showing it on the screen

...if it would boot just without output, I would be able to ping it and login - it appears hung.
...I havn't figured out this "earlycon" yet.

found ACPI_DEBUG=y in the distros that worked for me.

I have that already.


...in my many years of using Linux, I never had a problem with kernel just being black and hang.

Any ideas on how to debug this? - otherwise I see as only option to grab the Ubuntu kernel (config), try to PXE boot it, and if it shows output, then bisect it with my kernel config. And this takes days...

Ideas appreciated.

[pietinger]

mortonP,

unfortunately, I can't tell from your report whether you ‘only’ use our (preconfigured) distribution kernel, or also (manually) configured kernels. What you can try with any kernel is:
https://wiki.gentoo.org/wiki/User:Pieti ... and_others

However, you should check the following first:

1. Have you switched off SecureBoot in the bios?

2. Is this a machine with a BMC? If yes, please check:
https://wiki.gentoo.org/wiki/User:Pieti ... on.3Dmap:1

3. If you use a manually configured kernel, take advantage of Ubuntu booting and find out everything about this configuration; do at least these queries:
https://wiki.gentoo.org/wiki/User:Pieti ... _you_start
... and the complete output of "dmesg". Also save the kernel configuration of Ubuntu to compare it with yours later:

Code: Select all

zfgrep "" /proc/config.gz > ubuntu-kernel-x.y.z.config

[mortonP]

It is a vanilla 6.15.0 upstream kernel, because I thought newest hardware requires newest kernel.
The kernel config is a "grown" custom one, but has worked over years on all Intel and AMD systems without a problem.

Thank you VERY MUCH for the hint of
"earlycon=efifb efi=debug initcall_debug ignore_loglevel keep_bootcon"!
I wasted hours already rebooting and rebooting to a black screen!

Now I do see some output, kernel does boot, but hangs at some power management point in ACPI:
https://i.imgur.com/5ALl587.jpeg

Now the questions is what Ubuntu does different in their kernel options...

(and I don't know yet whether my Gentoo install is actually good, it hasn't booted so far yet...)

[mortonP]

> Now the questions is what Ubuntu does different in their kernel options...

https://i.imgur.com/hBvjHuD.jpeg

right is Ubuntu, left is mine ..... could it be one of these options?

...or is it a cmdline parameter?
...or some other option?

hmmm....

[NeddySeagoon]

mortonP,

A few things,

1. It's unlikely the line you want is on the screen, so it may not be ACPI at all.
2. It's difficult to work with filtered data. We can't see what should be there but is missing.

Pastebin both kernel .config files, so we can see it all.

Post the output of

Code: Select all

lspci -nnk

Post the output of

Code: Select all

lsusb

with any essential USB peripherals plugged in. e.g. mouse, keyboard ...
Neither of these are Gentoo specific.

Now we can look over your hardware and kernel.
A list of the filesystems you want to use may help too.

[mortonP]

I tried a build of Ubuntu's 6.14 sources with my config -> hang -> it's not the kernel source/version?
...points to that it is a kernel option.


My kernels get stuck at the D3C_ line - see photo

Ubuntu's dmesg is:
[ 0.618358] ACPI: \_SB_.PC00.TBT0: New power resource
[ 0.618590] ACPI: \_SB_.PC00.TBT1: New power resource
[ 0.618818] ACPI: \_SB_.PC00.D3C_: New power resource
[ 0.664864] ACPI: \PIN_: New power resource
[ 0.664903] ACPI: \PINP: New power resource
[ 0.665908] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-fe])
[ 0.665920] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[ 0.668459] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME AER PCIeCapability LTR DPC]

The PIN line would be the next one and is missing when boot hangs.

Google says:
`\PIN:\` represents a new ACPI resource type related to power management. Specifically, it's used to represent programmable power control resources such as voltage/current providers (regulators) and clock sources.

As this is a laptop, this would make sense...

[pietinger]

I don't know if you are checking it yourself now or if you still need help. If the latter, we need the same information as I already described in my last post (number 3) .. and of course your "broken" kernel .config.

[mortonP]

We have a winner! (by brute-force bi-secting of .config)

hangs:
# CONFIG_X86_X2APIC is not set

works:
CONFIG_X86_X2APIC=y

...why this was not set I don't know - as I said, this config has mutated over the years over many systems. But apparently, Arrowlake REALLY needs that option now enabled (and from reading the description, it seems to be a good thing to do that)

Thank you all for help - I hope everyone learned something like I did!

[pietinger]

CONFIG_X86_X2APIC=y

Yes ... this is a MUST HAVE for all modern Intel machines ... with 6.15 it is now enabled by default in the default config ... maybe see also:
* https://wiki.gentoo.org/wiki/User:Pieti ... rsion_6.12 - OR -
* https://wiki.gentoo.org/wiki/User:Pieti ... ent_Kernel

Thank you all for help

You are very Welcome !

[mortonP]

Oh, you seem to be a kernel configuration expert and keeping track of what problems users have - that's why you asked :-)

I try to keep a common kernel config across all machines, very old and very new, Intel and AMD, because being able to PXE boot into a recovery system on all if needed is SO useful.
However, this was the first time my "boot into PXE minimal to setup new hardware" strategy did not work - and I spent instead 2 days debugging this :-(

Will go through your guide when I find some time, to find other options that were added over the years and I never enabled and may be useful (e.g. that FRED option for newer hardware?)
Also, need to throw out more drivers for hardware that I'll never have - these kernel builds just get too big.

[pietinger]

Oh, you seem to be a kernel configuration expert and keeping track of what problems users have - that's why you asked

No, I am not a kernel expert ... I have only a little experience because I configure my kernels more than 20 years ... AND ... I learned a lot from others - like our Neddy - and from problems users had (and we searched a lot/long time). Yes, I watch the changes to every new (major) kernel version (see my articles in https://wiki.gentoo.org/wiki/User:Pieti ... perimental).

I try to keep a common kernel config across all machines, very old and very new, Intel and AMD, because being able to PXE boot into a recovery system on all if needed is SO useful. [...]

Will go through your guide when I find some time, [...]

My recommendations are focused on security and a slim kernel. Just ignore every comment where I ask if you have AMD or Intel ... and enable both (e.g. both LPSS; both P-state; both IOMMU; ...)

(e.g. that FRED option for newer hardware?)

Simply go to your newest and most modern machines and do a:

Code: Select all

# cpuid -1 | grep FRED
      FRED transitions & MSRs                  = false

This will answer it. (This is the answer from my old Intel i9-13900K) (I have a note about cpuid-queries also in my article)

Also, need to throw out more drivers for hardware that I'll never have - these kernel builds just get too big.

Yes, this is one of the big advantages we have when we configure our kernel ourself - we can throw out everything not needed

[pietinger]

[...] (e.g. that FRED option for newer hardware?)

Hmmm ... maybe wait for new kernel versions ...
https://www.phoronix.com/news/Intel-FRE ... le-ENDBR64

[penguinomicon]

There's a possibility which I didn't see mentioned yet, which is, could it be your dracut config? The reason I mention it is, I recently saw a similar problem to what OP describes, except it was in Void Linux: https://github.com/void-linux/void-pack ... sues/56691

Basically, unless you have hostonly=yes set, the initramfs keeps getting bigger over time, and once it hits a certain threshold the kernel will freeze while trying to load the initramfs.

I believe gentoo has better defaults, but if you tweaked them then you might experience the same symptoms.

Anyway, to rule this out I suggest you could boot from a liveusb, mount your root/boot disk, and check the size of the initramfs. If it's bigger than a couple of hundred mb then check your dracut config, make sure you have hostonly=yes, and trigger a rebuild of the initramfs.

Sorry if the above turns out to not be your problem, but I was bitten by it, so hopefully these are useful pointers for someone.