Kernel panic upon boot

[NeddySeagoon]

amosbarsinai,

This post from 2005 gives a general method.
At 21 years old, it's showing its age. The details are no longer correct.

You need to understand the PC Boot Process
The section Configuring the kernel is particularly important.

Further, you need to use your search engine of choice to search site:cateee.net for USB and PCI Vendor and Device IDs as it will tell you the lowest for level hardware driver that is required.
For example

Code: Select all

... 
09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
	Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
	Kernel driver in use: ahci
...

This is a PCI device as its listed in lspci. The Vendor and Device IDs are 1022:7901, from the end of the first line.
Putting 1022:7901 PCI site:cateee.net into my search enige gets me two hits

https://cateee.net/lkddb/web-lkddb/SATA_AHCI.html and https://cateee.net/lkddb/web-lkddb/ATA.html
Both are relevant.
Read the depends on: in the help. That's a boolean equation that must evaluate to true or the option will be hidden.
Now you know what symbol to find in the menuconfig search.

The Kernel driver in use: is a big help too.

Something similar mostly works for USB devices too. I say 'mostly' because some USB drivers are for a whole class of devices.

[amosbarsinai]

Uhhhh...
Might have figured out the problem.
I enabled the frame buffer thing and the console and all that, and...
now...
the reason I'm waiting is, according to Plymouth, a start job for ZFS pools?
I'm on Btrfs...
And the ZFS just keeps going:

Code: Select all

Finished Set BOOTFSS and BOOTFSFLAGS environment variables for dracut.
Reached target ZFS pool import target.
Finished dracut pre-mount hook.
A start job is running for /dev/disk/by-uuid/<my uuid> (7min 27s / no limit)

[NeddySeagoon]

amosbarsinai,

dracut is an initrd builder.
When you make your own kernel, you probably don't need an initrd. At least, not if you can build everything in to be able to mount root.

Are you using an initrd that you don't need, or are not expecting to need?

[amosbarsinai]

I don't know
I just emerged the sources, went into nconfig to change the things I needed, and ran make
make install executed dracut...

[NeddySeagoon]

amosbarsinai,

Why do you think you need the dracut step?
What is an initrd going to do for you?

[amosbarsinai]

I don't think I need an initrd
Like... maybe...
But I didn't choose to have one
make install just runs dracut and I don't mind...
Is it better to remove it?

[dmpogo]

I don't think I need an initrd
Like... maybe...
But I didn't choose to have one
make install just runs dracut and I don't mind...
Is it better to remove it?

I for one do not use intird, but accurate setup requires some understanding which of your hardware drivers require firmware

[NeddySeagoon]

amosbarsinai,

Assuming your boot loader is Grub, you can cheat to test.

Choose the kernel you want to boot from the menu then press 'e' for edit.
Remove the initrd line.
You can remove the echo statement too but that's just text.

Kernels that must boot anywhere need an initrd to hold all the possible modules that such a wide selection of hardware could have.
(Mostly) only used modules stay loaded.
When you build your own kernel wth everything built in to boot, it will only work on hardware identical to yours but no kernel modules are loaded from the initrd. Some arrangements need user space programs in an initrd but the simple case needs neither, so no initrd is requied.

Installkernel has a dracut USE flag.
Right now you need that on for binary kernels and off for your own kenel.
The workaround is to remove the initrd at boot time, until it works, then make your mind up.
Then again, you could not use installkernel for you own kernels.

[amosbarsinai]

So:
I nuked /boot, reinstalled sys-kernel/installkernel without the dracut flag, rebuilt and reinstalled my kernels (gentoo-sources and vanilla-sources), and the gentoo-sources boots faster than ever... like, a second, more or less (Windows took at least a minute to boot on my computer).
vanilla-sources gives a kernel panic (and a nice looking one - I enabled the DRM_PANIC flag )

Code: Select all

VFS: Unable to mount root fs on "/dev/sda3" or unkown-block(8,3)

[pietinger]

Code: Select all

[...] unkown-block(8,3)

If you have numbers here that means: Kernel has access to the harddisk and tries to mount a given partition (given via root=) .. but is unable to do that ... most reason for this: It is a filesystem kernel doesnt have ...

(or it is the wrong partition and kernel checks it is not the root partition; or you miss the partition type, e.g. CONFIG_EFI_PARTITION=y)

[pietinger]

P.S.:

amosbarsinai,

please note that my wiki articles on manual kernel configuration are based solely on our gentoo-sources (as stated [*]).

And it may of course be that some things are not activated in the default configuration of the vanilla-sources that are already activated by default in our gentoo-sources.

This means that I have not addressed anything that is already enabled by default. For example, both partition types are already enabled, as stated here:
https://wiki.gentoo.org/wiki/User:Pieti ... lock_layer
=>

Nothing else to do here, because in "Partition Types" both important options are already enabled (PC BIOS (MSDOS partition tables) and EFI GUID Partition support).

*) https://wiki.gentoo.org/wiki/User:Pieti ... ent_Kernel
=>

Only our default sys-kernel/gentoo-sources will be used.

[NeddySeagoon]

amosbarsinai,

That's progress in the right direction.

The (8,3) is the kernels major,minior device numbers for sda3.
As well as the causes that pietinger described, there is a more subtle one.
Drives are assigned kernel names in discovery order. That means that if you have two or more drives, the kernel names can change.

Use root=PARTUUID=... so that it doesn't matter. The kernel will sort the mess out.
If you have exactly one drive, you don't have this problem.

[amosbarsinai]

Thanks to you all!
I just fixed it myself... (obviously that's a blatant lie, I couldn't do it without each and every one of you )
I just baked Btrfs into the kernel - solved the problem.
Now it works, but everything is abysmally slow, internet just doesn't work, my mouse doesn't work -
but -
0.4 seconds boot

[NeddySeagoon]

amosbarsinai,

That's the hard bit done. It boots and you have a console.
Well done!

Now you can look at dmesg for errors.
That makes fixing it much easier than boot failures.

If you want to show us dmesg, you need to put it on a pastebin site as it's too big for a post:
wgetpaste can help. You may need to use the -S and -s options.

[pietinger]

The (8,3) is the kernels major,minior device numbers for sda3.

amosbarsinai,

if you have questions about this then look into /usr/src/linux/Documentation/admin-guide/devices.txt (and scroll down to "8 block").

[dmpogo]

*) https://wiki.gentoo.org/wiki/User:Pieti ... ent_Kernel
=>

Only our default sys-kernel/gentoo-sources will be used.

That is an amazing guide and work !

I am coming from years of compiling EFI stub 5.15 kernels for my current machine, and is using your guide extensively goign for 6.17 for my new laptop while runnoing 6.12.47 distribution-bin in the meantime.
But I have some questions with firmwares (which prolifirates ) and microcode.

If I emerge intel-microcode with USE='hostonly" it instsalls three compatible files for my CPU (Intel Meteor Lake) 06-aa-01, 06-aa-02, and 06-aa-04

My CPU ID is family 6 (so fsar so good), model 170 (this does translate into HEX aa), and stepping 4. Does it mean I need only the last -04 microcode, or all three ?
Your discussion points at only one, 06-aa-04, right ?


Sadly I cannot see anywhere what microcode is loaded by distribution kernel

[pietinger]

[...] Does it mean I need only the last -04 microcode, or all three ?
Your discussion points at only one, 06-aa-04, right ?

Yes, you need only this one file.

[...] Sadly I cannot see anywhere what microcode is loaded by distribution kernel

You should see in your syslog two lines ... this is mine:

Code: Select all

# dmesg | grep microcode
[    8.794758] microcode: Current revision: 0x0000012f
[    8.794759] microcode: Updated early from: 0x00000113

Sometimes, you will see only one line ... like in this thread: https://forums.gentoo.org/viewtopic-p-8 ... ml#8873253 ->

Code: Select all

microcode: Current revision: 0x00000121

Then there exists two reasons for it:
1. The microcode was not loaded beacuse it was not accessible (you have two ways to give the kernel the microcode-file: Built-in with CONFIG_EXTRA_FIRMWARE=... -OR- as part of the initramfs), -OR-
2. The current microcode is already available in the UEFI (BIOS) (if you have a very new machine) and has already been loaded by the CPU. Since there is no newer version available, no newer version can be loaded... just wait for a new microcode version so that you get two lines.

[pietinger]

That is an amazing guide and work !

Thank you very much for this compliment!

[pietinger]

P.S.:

dmpogo,

you can always look into https://github.com/intel/Intel-Linux-Pr ... asenote.md

Last Update for your CPU (06-aa-04) was with microcode-20250812 to version 00000025.

[dmpogo]

P.S.:

dmpogo,

you can always look into https://github.com/intel/Intel-Linux-Pr ... asenote.md

Last Update for your CPU (06-aa-04) was with microcode-20250812 to version 00000025.

Thanks, indeed dmesg says on boot that microcode was updated form 00024 to to 000025

[pietinger]

dmpogo,

Just a quick note: I updated my page about KSPP extensively last night because Kees Cook also updated his page [*] yesterday:
https://wiki.gentoo.org/wiki/User:Pieti ... _with_KSPP

*) https://kspp.github.io/Recommended_Settings

[dmpogo]

dmpogo,

Just a quick note: I updated my page about KSPP extensively last night because Kees Cook also updated his page [*] yesterday:
https://wiki.gentoo.org/wiki/User:Pieti ... _with_KSPP

*) https://kspp.github.io/Recommended_Settings

Thanks ! All info comes handy !

But currently I go opposite way - it is a laptop, so minimum security, no hardening, no encryption, even no CPU vulnerabilitity mitigations

[dmpogo]

Menuconfig is really frustrating with 6.x.x kernels (where are my 5.15) There seems to be plenty of loops where things select each other, or strange restriction. For instance currently I failed to both unset DEBIUG_FS and choose only Intel CPU,
since one requires EXPERT=y and the other EXPERT=n. That feels totally ridiculous

[pietinger]

Menuconfig is really frustrating with 6.x.x kernels (where are my 5.15) There seems to be plenty of loops where things select each other, or strange restriction. For instance currently I failed to both unset DEBIUG_FS and choose only Intel CPU,
since one requires EXPERT=y and the other EXPERT=n. That feels totally ridiculous

No ... You can enable EXPERT and disable DEBUG_FS ... (I have both options configured)

To be able to disable DEBUG_FS you must disable BLK_DEV_IO_TRACE [1] ... I have described it here:
https://wiki.gentoo.org/wiki/User:Pieti ... el_hacking

1) In the <Help> of DEBUG_FS you can see:

Code: Select all

Selected by
...
BLK_DEV_IO_TRACE [=n] && FTRACE [=n] && SYSFS [=y] && BLOCK [=y]

There is only ONE line which has EXPERT in the "Selected by" section ... and here you must disable I915_DEBUG (which is usually disabled by default) ... Do NOT disable EXPERT->

Code: Select all

Selected by
...
DRM_I915_DEBUG [=n] && HAS_IOMEM [=y] && DRM [=y] && DRM_I915 [=y] && EXPERT [=y] && !COMPILE_TEST [=n]

[dmpogo]

Menuconfig is really frustrating with 6.x.x kernels (where are my 5.15) There seems to be plenty of loops where things select each other, or strange restriction. For instance currently I failed to both unset DEBIUG_FS and choose only Intel CPU,
since one requires EXPERT=y and the other EXPERT=n. That feels totally ridiculous

No ... You can enable EXPERT and disable DEBUG_FS ... (I have both options configured)

To be able to disable DEBUG_FS you must disable BLK_DEV_IO_TRACE [1] ... I have described it here:
https://wiki.gentoo.org/wiki/User:Pieti ... el_hacking

1) In the <Help> of DEBUG_FS you can see:

Code: Select all

Selected by
...
BLK_DEV_IO_TRACE [=n] && FTRACE [=n] && SYSFS [=y] && BLOCK [=y]

There is only ONE line which has EXPERT in the "Selected by" section ... and here you must disable I915_DEBUG (which is usually disabled by default) ... Do NOT disable EXPERT->

Code: Select all

Selected by
...
DRM_I915_DEBUG [=n] && HAS_IOMEM [=y] && DRM [=y] && DRM_I915 [=y] && EXPERT [=y] && !COMPILE_TEST [=n]

That is embarasing, I meant DEBUG_KERNEL (forced by EXPERT), not DEBUG_FS




And regarding the other thread - I specifically wanted to split from this thread, which I kind of hijacked