Kernel panics on boot due to misconfigured LUKS setup.

[leyvi]

This is my kernel configuration.
The tty says something along the lines of "can't find root filesystem", and I suspect it might be due to my kernel arguments (which I set in the configuration file above, since I boot directly out of EFI, which doesn't always handle arguments correctly), which I don't entirely understand.
I've tried booting multiple times, most of the time it just panics, but once it actually prompted me for my root filesystem's passphrase (only for like half a second though, then it panicked).

Anyone know how to fix this?

[grknight]

One problem is that CONFIG_BLK_DEV_INITRD is not set in the kernel config.

LUKS needs an initramfs to unlock and prepare the rootfs if contained within.

Oh, and make sure you have created a suitable initramfs and referenced it in your loader, UKI or other kernel options

[leyvi]

One problem is that CONFIG_BLK_DEV_INITRD is not set in the kernel config.

LUKS needs an initramfs to unlock and prepare the rootfs if contained within.

Oh, and make sure you have created a suitable initramfs and referenced it in your loader, UKI or other kernel options

Thanks.
I had heard that no initramfs was needed, unless one wanted to use a keyfile (this is something I read in a post a while ago, someone suggested that's how their setup worked).
What exactly does it use the initramfs for?

[Hu]

I think the post you allude to was either wrong or you misunderstood it. Without a link back to the post, I cannot determine which applies.

The kernel has no innate ability to unlock a LUKS volume. The initramfs is a way, and likely the easiest way, to run the requisite userspace code to unlock the LUKS volume, mount the contained filesystem, and transfer control to it so that booting can continue.

[leyvi]

The post is quite old, old enough that it isn't showing up anymore, unless I misunderstand how this forum works.

This Arch Wiki page (this specific section) also suggests that decrypting in early userspace is possible without an initramfs, that is using only kernel arguments. Though maybe I misunderstood it...

[Hu]

This forum never automatically deletes posts, though posts beyond a certain age may be difficult to find with the built-in search. If you still have it in your browser history, the URL should be valid for you and can be shared here.

That page seems to discuss using early userspace, which is how some people refer to an initramfs. I admit I did not read the entire document, but of the parts I did read, I cannot find anything that seems to support your interpretation. The Arch Wiki page seems to expect users to use one of the pre-written tools that, through some simple configuration parameters, will handle the details of invoking cryptsetup. I did not see anything suggesting that you could create the dm-crypt device directly from the kernel command line. Even if you could, I expect you would need to specify the key on the kernel command line, which would be insecure. Everything under "1. Unlocking in early userspace" seems to be about configuring the Arch initramfs. Could you quote the specific passage that you think supports your position?

[pingtoo]

This Arch Wiki page (this specific section) also suggests that decrypting in early userspace is possible without an initramfs, that is using only kernel arguments. Though maybe I misunderstood it...

Actually the top of page explicitly stated that

Booting an encrypted root volume requires that the initramfs contains the necessary tools for early userspace to unlock the volume. The instructions on what to unlock are typically passed via kernel parameters.

IMHO Arch Wiki page are very detail but may be too much information all at once especially with all those links that lead to sometime too much that one forgot what was the initial focus.

[leyvi]

Ah, thanks for clarifying.
I've heard of dracut and uGRD, and I'm thinking the latter might be better for my use case (all I need from it is the LUKS stuff, firmware is already built-in).
Which do you recommend?

[pietinger]

I've heard of dracut and uGRD, and I'm thinking the latter might be better for my use case [...]
Which do you recommend?

The developer of ugrd @zen_desu is very active in our forum ...

Internal Note (because the pastebin service deletes the kernel .config after a certain amount of time):

Code: Select all

1.
# CONFIG_X86_INTEL_LPSS is not set
CONFIG_X86_AMD_PLATFORM_DEVICE=y
2.
# CONFIG_X86_NATIVE_CPU is not set
3.
CONFIG_MZEN4=y
4.
CONFIG_CMDLINE="cryptdevice=PARTUUID=5c98ed25-6874-46e7-83ac-36c670bdb669:rootfs root=/dev/mapper/rootfs resume=/swap/swapfile hugepagesz=1GB hugepages=8 pcie_aspm=force"
5.
CONFIG_BSD_DISKLABEL=y
CONFIG_EFI_PARTITION=y

CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y

# CONFIG_SCSI is not set

# CONFIG_ATA is not set

CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=y

CONFIG_DM_CRYPT=y

# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed; see USB_STORAGE Help for more info

CONFIG_TYPEC=y

CONFIG_EXT4_FS=m
CONFIG_XFS_FS=m
CONFIG_BTRFS_FS=y
CONFIG_F2FS_FS=m

1. It is a AMD machine
2. kernel version 6.17.1 has this option (since 6.16) ... and with this new option (*) there is no need to emerge gentoo-sources ...
3. ... with USE-flag "experimental"
4. ...
5. A machine without SCSI ... never able to read an USB-stick (yes, also USB-C needs SCSI if there is a need to use CONFIG_USB_STORAGE)... root FS seems to be BTRFS ... booting from a NVMe ... no SATA/AHCI module enabled

*) see more:
- https://www.phoronix.com/news/Linux-6.16-X86_NATIVE_CPU
- https://wiki.gentoo.org/wiki/User:Pieti ... 15_to_6.16

[leyvi]

I've heard of dracut and uGRD, and I'm thinking the latter might be better for my use case [...]
Which do you recommend?

The developer of ugrd @zen_desu is very active in our forum ...

Alright, so I'll use that
I messed around with it for a bit, it should be pretty easy.

On a separate note:
I use secure boot to verify my kernel, how do I verify that the initramfs hasn't been tampered with? I realize that this is more of a security question that a kernel/hardware question, so if you think I should post this question on the appropriate part of the forum, I'll happily do that.

[zen_desu]

I've heard of dracut and uGRD, and I'm thinking the latter might be better for my use case [...]
Which do you recommend?

The developer of ugrd @zen_desu is very active in our forum ...

Alright, so I'll use that
I messed around with it for a bit, it should be pretty easy.

On a separate note:
I use secure boot to verify my kernel, how do I verify that the initramfs hasn't been tampered with? I realize that this is more of a security question that a kernel/hardware question, so if you think I should post this question on the appropriate part of the forum, I'll happily do that.

you more or less need to bundle the initramfs into the signed image, so you can use a UKI or this: https://github.com/desultory/ugrd/blob/ ... the-kernel

[leyvi]

What is a UKI?

[zen_desu]

What is a UKI?

unified kernel image, a single file which has the kernel, initramfs, microcode, cmdline and is directly bootable on EFI systems

[leyvi]

What do you recommend:

1. Building the initramfs into the kernel.
2. Making a UKI (however that is done, I have no clue).

[zen_desu]

What do you recommend:

1. Building the initramfs into the kernel.
2. Making a UKI (however that is done, I have no clue).

Building it into the kernel is a bit trickier, especially if modules end up being needed.

If you can setup your kernel so modules are not needed, you can build an initramfs once and use that until the userspace components within it actually need to be updated.

The trouble comes from the fact that when the kernel builds the initramfs in, it has to exist before the kernel finished building, so if modules are used, you'll have to: build the kernel, install modules, build an initramfs, rebuild the kernel.
UKIs help because they let you pack things up after the fact. Both can do about the same thing so it largely comes down to preference. I prefer not to use UKIs that much because you need systemd tools/parts to make them work (at this point in time)

[leyvi]

Thanks zen_desu!

[pietinger]

leyvi,

with Gentoo Linux you have always two options:

1. Create something with "automagic" routines (where you don't know who does what and how), or
2. create something yourself with the tools you know, so you know how you created it.

[...] I prefer not to use UKIs that much because you need systemd tools/parts to make them work (at this point in time)

This statement is correct ... for the first case.

If you google "linux uki" you will get these three links on the first results page:

a) https://wiki.archlinux.org/title/Unified_kernel_image
->

A unified kernel image (UKI) is a single executable which can be booted directly from UEFI firmware, or automatically sourced by boot loaders with little or no configuration. It is the combination of a UEFI boot stub program like systemd-stub(7), a Linux kernel image, an initramfs, and further resources in a single UEFI PE file.

I emphasize: "It is the combination of a UEFI boot stub program like systemd-stub."

b) https://wiki.gentoo.org/wiki/Unified_kernel_image
->

A unified kernel image (UKI) is a single executable which can be booted directly from UEFI firmware, or automatically sourced by boot-loaders with little or no configuration.

A unified kernel image allows to incorporate all or a subset of the following:

an EFI stub loader like systemd-stub,

[...]
I emphasize: "an EFI stub loader like systemd-stub"

c) https://uapi-group.org/specifications/s ... nel_image/
->

A Unified Kernel Image (UKI) is a combination of an UEFI boot stub program, a Linux kernel image, an optional initrd, and further resources in a single UEFI PE file. [...]

systemd-stub provides the reference implementation of the stub.

...

The plain truth is that every Linux kernel already has a EFI stub (if you activate it in the kernel configuration (*) - this is the default in all our Gentoo distribution kernels) and you don't need this systemd-stub at all ... as long as you do everything yourself ... it's just that many "automagic" routines were written by people who only work with systemd ...

*) =>

Code: Select all

Processor type and features  --->
    [*]   EFI stub support

The <Help> of this option says it clearly:

This kernel feature allows a bzImage to be loaded directly by EFI firmware without the use of a bootloader.

Yes, I have an "UKI" myself, even if it cannot be described as such according to the "academic" interpretation — because it does not have a systemd stub. I have a signed (for SecureBoot) kernel image that contains an embedded initramfs, the kernel command line, and the microcode for the CPU (and also all needed firmware binaries). Even though I use a monolithic kernel (= without module support because everything is configured as built-in), this would not be a mandatory requirement. The only requirement for such an image is that you configure the kernel modules that the kernel needs to address its root partition as built-in (this means you can configure the graphics card modules or sound modules as <M>odule). Then you no longer have to take this "detour" via: build the kernel, install modules, build an initramfs, rebuild the kernel. All you need to do is build the kernel and install the modules. If you even configure a monolithic kernel (like I do), all you need to do is: BUILD KERNEL. (The end, amen, done).

And yes, creating your own initramfs is not that difficult:
https://wiki.gentoo.org/wiki/User:Pieti ... s_Overview
(Like @zen_desu, I also have instructions on how to insert an existing external CPIO initramfs into the kernel:
https://wiki.gentoo.org/wiki/User:Pieti ... IO_archive )

I use an activated IMA (ima_appraise=enforce) with signed hashes on my notebook, where I have to load the certificate and policy into the kernel. Look how short the init routine is for this: https://forums.gentoo.org/viewtopic-t-1159297.html

Of course, it takes a considerable amount of time to do everything yourself... but in return, you know what's going on in your system... and you also learn a lot... which you can then use to understand automatic routines.

[leyvi]

zen_desu, does µgRD have plymouth support?

[leyvi]

Also:
My current configuration (for µgRD) doesn't work for some reason:

Code: Select all

CRITICAL | '/rootfs'
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/ugrd/main.py", line 175, in main
    generator.build()
    ~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/ugrd/initramfs_generator.py", line 99, in build
    self.run_checks()
    ~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/ugrd/initramfs_generator.py", line 278, in run_checks
    if check_output := self.run_hook("checks"):
                       ~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/ugrd/initramfs_generator.py", line 162, in run_hook
    if function_output := self.run_func(function, *args, **kwargs):
                          ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/ugrd/initramfs_generator.py", line 110, in run_func
    if function_output := function(self):
                          ~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/site-packages/zenlib/util/dict_check.py", line 36, in _contains
    return func(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/ugrd/fs/mounts.py", line 953, in check_mounts
    _validate_host_mount(self, mount, "/" if mount_name == "root" else None)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/ugrd/fs/mounts.py", line 909, in _validate_host_mount
    host_source_dev = _resolve_dev(self, self["_mounts"][destination_path]["device"])
                                         ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: '/rootfs'

Do you happen to know what the issue is, or rather, how to fix it?

[leyvi]

Code: Select all

[mounts.rootfs]
# The label of the root filesystem, a uuid or partuuid could be used instead
label = "rootfs"

I'm guessing this is where the issue is; I'm trying to mount the root filesystem, which should be named /dev/mapper/rootfs, can I give it that path instead of a label?

[zen_desu]

Code: Select all

[mounts.rootfs]
# The label of the root filesystem, a uuid or partuuid could be used instead
label = "rootfs"

I'm guessing this is where the issue is; I'm trying to mount the root filesystem, which should be named /dev/mapper/rootfs, can I give it that path instead of a label?

you shouldn't need to set mounts like that, but it will validate mounts against your current mounts, so unless you have something mounted at "/rootfs", it won't pass validation.

Your root mount should be automatically detected, so I think the default config should be mostly fine. Users can also specify paths to be automatically processed using "auto_mounts = ['/patha', '/pathb']" This is mostly useful if you want something like /boot to be mounted or have a keyfile device that needs to be mounted before cryptsetup runs.

If you need to set special config for a particular cryptsetup device (adding a key file/headers/other options) you can set it under [cryptsetup.<devname>] so in your case, [cryptsetup.rootfs]

The "mounts" are specifically for things to be mounted in the initramfs stage, and the root mount is internally called "[mounts.root]" and uses the path "/target_rootfs" in the initramfs: https://github.com/desultory/ugrd/blob/ ... ml#L76-L79

Unless you disable hostonly mode, or set "autodetect_root = false", it should automatically get this info using the current "/" mount: https://github.com/desultory/ugrd/blob/ ... ts.py#L715

[leyvi]

leyvi,

with Gentoo Linux you have always two options:

1. Create something with "automagic" routines (where you don't know who does what and how), or
2. create something yourself with the tools you know, so you know how you created it.

[...] I prefer not to use UKIs that much because you need systemd tools/parts to make them work (at this point in time)

This statement is correct ... for the first case.

If you google "linux uki" you will get these three links on the first results page:

a) https://wiki.archlinux.org/title/Unified_kernel_image
->

A unified kernel image (UKI) is a single executable which can be booted directly from UEFI firmware, or automatically sourced by boot loaders with little or no configuration. It is the combination of a UEFI boot stub program like systemd-stub(7), a Linux kernel image, an initramfs, and further resources in a single UEFI PE file.

I emphasize: "It is the combination of a UEFI boot stub program like systemd-stub."

b) https://wiki.gentoo.org/wiki/Unified_kernel_image
->

A unified kernel image (UKI) is a single executable which can be booted directly from UEFI firmware, or automatically sourced by boot-loaders with little or no configuration.

A unified kernel image allows to incorporate all or a subset of the following:

an EFI stub loader like systemd-stub,

[...]
I emphasize: "an EFI stub loader like systemd-stub"

c) https://uapi-group.org/specifications/s ... nel_image/
->

A Unified Kernel Image (UKI) is a combination of an UEFI boot stub program, a Linux kernel image, an optional initrd, and further resources in a single UEFI PE file. [...]

systemd-stub provides the reference implementation of the stub.

...

The plain truth is that every Linux kernel already has a EFI stub (if you activate it in the kernel configuration (*) - this is the default in all our Gentoo distribution kernels) and you don't need this systemd-stub at all ... as long as you do everything yourself ... it's just that many "automagic" routines were written by people who only work with systemd ...

*) =>

Code: Select all

Processor type and features  --->
    [*]   EFI stub support

The <Help> of this option says it clearly:

This kernel feature allows a bzImage to be loaded directly by EFI firmware without the use of a bootloader.

Yes, I have an "UKI" myself, even if it cannot be described as such according to the "academic" interpretation — because it does not have a systemd stub. I have a signed (for SecureBoot) kernel image that contains an embedded initramfs, the kernel command line, and the microcode for the CPU (and also all needed firmware binaries). Even though I use a monolithic kernel (= without module support because everything is configured as built-in), this would not be a mandatory requirement. The only requirement for such an image is that you configure the kernel modules that the kernel needs to address its root partition as built-in (this means you can configure the graphics card modules or sound modules as <M>odule). Then you no longer have to take this "detour" via: build the kernel, install modules, build an initramfs, rebuild the kernel. All you need to do is build the kernel and install the modules. If you even configure a monolithic kernel (like I do), all you need to do is: BUILD KERNEL. (The end, amen, done).

And yes, creating your own initramfs is not that difficult:
https://wiki.gentoo.org/wiki/User:Pieti ... s_Overview
(Like @zen_desu, I also have instructions on how to insert an existing external CPIO initramfs into the kernel:
https://wiki.gentoo.org/wiki/User:Pieti ... IO_archive )

I use an activated IMA (ima_appraise=enforce) with signed hashes on my notebook, where I have to load the certificate and policy into the kernel. Look how short the init routine is for this: https://forums.gentoo.org/viewtopic-t-1159297.html

Of course, it takes a considerable amount of time to do everything yourself... but in return, you know what's going on in your system... and you also learn a lot... which you can then use to understand automatic routines.

Completely missed this before, very interesting... thanks pietinger!

[leyvi]

Code: Select all

[mounts.rootfs]
# The label of the root filesystem, a uuid or partuuid could be used instead
label = "rootfs"

I'm guessing this is where the issue is; I'm trying to mount the root filesystem, which should be named /dev/mapper/rootfs, can I give it that path instead of a label?

you shouldn't need to set mounts like that, but it will validate mounts against your current mounts, so unless you have something mounted at "/rootfs", it won't pass validation.

Your root mount should be automatically detected, so I think the default config should be mostly fine. Users can also specify paths to be automatically processed using "auto_mounts = ['/patha', '/pathb']" This is mostly useful if you want something like /boot to be mounted or have a keyfile device that needs to be mounted before cryptsetup runs.

If you need to set special config for a particular cryptsetup device (adding a key file/headers/other options) you can set it under [cryptsetup.<devname>] so in your case, [cryptsetup.rootfs]

The "mounts" are specifically for things to be mounted in the initramfs stage, and the root mount is internally called "[mounts.root]" and uses the path "/target_rootfs" in the initramfs: https://github.com/desultory/ugrd/blob/ ... ml#L76-L79

Unless you disable hostonly mode, or set "autodetect_root = false", it should automatically get this info using the current "/" mount: https://github.com/desultory/ugrd/blob/ ... ts.py#L715

Thanks, no errors at runtime, gotta see if it boots...

[leyvi]

Well, it almost got there...

First try there were some issues with the HID modules, so I built those into the kernel (the ones that were giving error messages) and the errors went away.
Everything looks fine, that is, it prompts me for the disk's passphrase, but after I type it and press <Enter>, nothing happens, no matter how long I wait.

What now?
I can't check dmesg since I doubt that the root filesystem got mounted...

[zen_desu]

Well, it almost got there...

First try there were some issues with the HID modules, so I built those into the kernel (the ones that were giving error messages) and the errors went away.
Everything looks fine, that is, it prompts me for the disk's passphrase, but after I type it and press <Enter>, nothing happens, no matter how long I wait.

What now?
I can't check dmesg since I doubt that the root filesystem got mounted...

which HID modules did you need to build in?

The prompt looks like a cryptsetup prompt right? Do you have any args like "quiet" added to your kernel command line?

It sounds like cryptsetup is hanging which is abnormal. Generally if it's missing parts it'll log failure info to the console. There is a small chance it's running out of memory running the KDF but it will also typically warn about this and not just stall.