[solved] dracut quietly fails to install ZFS kernel modules

[kwesadilo]

I'm using gentoo-kernel and building it into a UKI via dracut. My rootfs is on ZFS, so I have ZFS support configured in dracut. In the ordinary course of updating packages, zfs-kmod will be installed before gentoo-kernel, and my UKI will have ZFS support and boot. This is my first time setting up a distribution kernel and some other things, so I've been reinstalling gentoo-kernel a lot. When I do, it fails to install the ZFS kernel modules in my UKI, but the emerge command still exits with successful status:

Code: Select all

# emerge gentoo-kernel

...

Reading /usr/lib/kernel/install.conf...
/usr/lib/kernel/install.conf configures layout=uki
/usr/lib/kernel/install.conf configures initrd_generator=dracut
/usr/lib/kernel/install.conf configures uki_generator=dracut

Running /usr/lib/kernel/preinst.d/35-amd-microcode.install 6.12.25-gentoo-dist-hardened /usr/src/linux-6.12.25-gentoo-dist-hardened/arch/x86/boot/bzImage...

initrd_generator=dracut bundles CPU microcode, nothing to do here.

Hook /usr/lib/kernel/preinst.d/35-amd-microcode.install finished successfully


Running /usr/lib/kernel/preinst.d/50-dracut.install 6.12.25-gentoo-dist-hardened /usr/src/linux-6.12.25-gentoo-dist-hardened/arch/x86/boot/bzImage...

 [1;32m*[0m Using dracut as the initramfs and UKI generator...
dracut[I]: Executing: /usr/bin/dracut --add-confdir hostonly --kernel-image /usr/src/linux-6.12.25-gentoo-dist-hardened/arch/x86/boot/bzImage --verbose --uefi /var/tmp/portage/sys-kernel/gentoo-kernel-6.12.25/temp/installkernel.staging.hJsIFz4/uki.efi 6.12.25-gentoo-dist-hardened
dracut[D]: Module 'dash' will not be installed, because command 'dash' could not be found!

...

dracut[I]: *** Including module: kernel-modules ***
dracut[I]: *** Including module: kernel-modules-extra ***
dracut[D]:   kernel-modules-extra: configuration source "/run/depmod.d" does not exist
dracut[D]:   kernel-modules-extra: configuration source "/etc/depmod.d" does not exist
dracut[D]:   kernel-modules-extra: configuration source "/lib/depmod.d" does not exist
dracut[I]: *** Including module: mdraid ***
dracut[I]: *** Including module: zfs ***
dracut[I]: *** Including module: rootfs-block ***
dracut[I]: *** Including module: terminfo ***
dracut[I]: *** Including module: udev-rules ***
dracut[I]: *** Including module: usrmount ***
dracut[I]: *** Including module: base ***
dracut[I]: *** Including module: fs-lib ***
dracut[I]: *** Including module: shell-interpreter ***
dracut[I]: *** Including module: shutdown ***
dracut[I]: *** Including modules done ***
dracut-install: Failed to find module 'spl'
dracut[E]: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/portage/sys-kernel/gentoo-kernel-6.12.25/temp/dracut.ZxypCw/initramfs --kerneldir /lib/modules/6.12.25-gentoo-dist-hardened/ -m spl zfs amdgpu
dracut[I]: *** Installing kernel module dependencies ***
dracut[I]: *** Installing kernel module dependencies done ***
dracut[I]: *** Resolving executable dependencies ***
dracut[I]: *** Resolving executable dependencies done ***
dracut[I]: *** Hardlinking files ***

...

dracut[D]: dracut cmdline:
dracut[D]: rd.md.uuid=18286aed:0f58aa00:7946ca1d:4fbf24d3
dracut[D]: root=zfs:tank/rootfs rootfstype=zfs rootflags=rw,relatime,xattr,posixacl,casesensitive
dracut[I]: Using UEFI kernel cmdline:
dracut[I]:  fbcon=nodefer  rd.md.uuid=18286aed:0f58aa00:7946ca1d:4fbf24d3  module.sig_enforce=1  root=zfs:tank/rootfs rootfstype=zfs rootflags=rw,relatime,xattr,posixacl,casesensitive 
Signing Unsigned original image
dracut[I]: *** Creating signed UEFI image file '/var/tmp/portage/sys-kernel/gentoo-kernel-6.12.25/temp/installkernel.staging.hJsIFz4/uki.efi' done ***

Hook /usr/lib/kernel/preinst.d/50-dracut.install finished successfully


Running /usr/lib/kernel/preinst.d/99-check-diskspace.install 6.12.25-gentoo-dist-hardened /usr/src/linux-6.12.25-gentoo-dist-hardened/arch/x86/boot/bzImage...

 [1;32m*[0m Checking available disk space on /efi/EFI/Linux...
 [1;32m*[0m Disk space okay. Need at least 10201 KiB, found 892888 KiB.

Hook /usr/lib/kernel/preinst.d/99-check-diskspace.install finished successfully

Found UKI directory on ESP /efi
Installing Unified Kernel Image for 6.12.25-gentoo-dist-hardened...

Running /etc/kernel/postinst.d/50-eclean-kernel.install 6.12.25-gentoo-dist-hardened /efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened.efi...

 [1;32m*[0m Removing old kernels ...
No outdated kernels found.

Hook /etc/kernel/postinst.d/50-eclean-kernel.install finished successfully


Running /usr/lib/kernel/postinst.d/95-efistub-uefi-mkconfig.install 6.12.25-gentoo-dist-hardened /efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened.efi...

 [1;32m*[0m Updating UEFI configuration...
 [1;32m*[0m Running uefi-mkconfig...
 [1;32m*[0m Using kernel commands from "/etc/default/uefi-mkconfig"
 [1;33m*[0m Warning! Kernel command "root=" is missing from loaded configuration!
 [1;32m*[0m Creating UEFI entry "0200" for "/efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened-old.efi" found on "md127p1"
 [1;33m*[0m No initramfs found for "/efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened-old.efi".
 [1;32m*[0m Creating UEFI entry "01FF" for "/efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened.efi" found on "md127p1"
 [1;33m*[0m No initramfs found for "/efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened.efi".
 [1;32m*[0m Done

Hook /usr/lib/kernel/postinst.d/95-efistub-uefi-mkconfig.install finished successfully


Running /usr/lib/kernel/postinst.d/99-write-log.install 6.12.25-gentoo-dist-hardened /efi/EFI/Linux/gentoo-6.12.25-gentoo-dist-hardened.efi...

 [1;32m*[0m Appending installed kernel to /var/log/installkernel.log...

Hook /usr/lib/kernel/postinst.d/99-write-log.install finished successfully

[A[152C [34;01m[ [32;01mok[34;01m ][0m
>>> sys-kernel/gentoo-kernel-6.12.25 merged.

dracut-install can't find spl.ko. I can reproduce this by running the dracut-install command directly. If I switch the order of the arguments, it will fail on zfs.ko instead. amdgpu.ko has no problem. All of these files exist in /lib/modules/6.12.25-gentoo-dist-hardened/, so I don't know why this command fails.

However, dracut and the overall merge claim to be successful. Without looking in the logs, there's no way to tell that something went wrong until I reboot, and the most recent kernel doesn't have zfs.ko and can't mount my rootfs.

I don't have this problem if I only run

Code: Select all

# emerge --config gentoo-kernel
...
dracut[I]: *** Including module: zfs ***
...
dracut[D]: -rw-r--r--   1 root     root       316262 May 15 08:39 usr/lib/modules/6.12.25-gentoo-dist-hardened/extra/spl.ko
dracut[D]: -rw-r--r--   1 root     root     12071398 May 15 08:39 usr/lib/modules/6.12.25-gentoo-dist-hardened/extra/zfs.ko
...

I get the same good-looking dracut run and bootable kernel when I emerge zfs-kmod. I get the same result the first time I emerge gentoo-kernel after emerging zfs-kmod. Subsequent emerge --config gentoo-kernel produces the same result. Subsequent full emerge gentoo-kernel fails as described above.

This feels like a bug in something, but I'm not familiar enough with the distribution kernel stack to know what. It seems to me like dracut-install should be able to find zfs.ko and spl.ko when they're in /usr/lib/modules, and it feels like the gentoo-kernel mere should fail if dracut-install fails. I don't know what changes when I emerge zfs-kmod. I can file a bug if need be, but I wanted to start here in case I'm missing something dumb.

[Hu]

You say you can reproduce the failure by running dracut-install args manually. In the case that dracut fails to install a kernel module, what is the exit code of the dracut process? If it is exiting with a success status code, then as far as emerge can tell, dracut succeeded. If dracut is exiting with a failure status code and the ebuild ignores that code, then that would be an ebuild problem.

[Nowa]

Code: Select all

emerge @module-rebuild

Will probably solve your problem.

Please check if you have the "dist-kernel" flag enabled, it should automatically trigger the module rebuild and will trigger initramfs regeneration in the zfs package.

If it is exiting with a success status code, then as far as emerge can tell, dracut succeeded. If dracut is exiting with a failure status code and the ebuild ignores that code, then that would be an ebuild problem.

Dracut intentionally treats these problems as non-fatal (yes that is very annoying). I've been working on an upstream solution for this specifically, but we are still stuck there: https://github.com/dracut-ng/dracut-ng/pull/1215

[kwesadilo]

I do have USE=dist-kernel enabled, and when I take an update to gentoo-kernel during emerge -uD @world, zfs-kmod gets rebuilt after gentoo-kernel. I take it that this is USE=dist-kernel doing its job. When I do something like emerge gentoo-kernel, it only rebuilds that one package and doesn't pull in modules. That would explain why I've only had this problem on a rebuild.

Is this how USE=dist-kernel is supposed to work?

I don't quite get the dracut-install failure mode in the kernel-rebuild case. I already installed zfs-kmod for the current kernel version, and zfs.ko is right there in file system, next to the in-tree modules that dracut-install finds.

Does dracut-install do some kind of build-ID check to reject kernel modules that don't match the exact target kernel image?

Dracut intentionally treats these problems as non-fatal (yes that is very annoying). I've been working on an upstream solution for this specifically, but we are still stuck there: https://github.com/dracut-ng/dracut-ng/pull/1215

Following, and I can weigh in on GitHub if that would be helpful. IIUC, this isn't really specific to zfs, though. I am currently installing amdgpu to my initramfs. If I wanted to do the same thing with nvidia-drivers, that would fail in the same way for the same reason. The resulting initramfs might boot, but the contents would be wrong. If that's correct, it would suggest that the fix should be general to out-of-tree modules, not just specific to zfs.

It sounds like the canonical Gentoo solution to this problem is USE=dist-kernel and @module-rebuild, and dracut upstream is the right place to talk about somehow handling this failure more coherently, at least for now.

[zen_desu]

I do have USE=dist-kernel enabled, and when I take an update to gentoo-kernel during emerge -uD @world, zfs-kmod gets rebuilt after gentoo-kernel. I take it that this is USE=dist-kernel doing its job. When I do something like emerge gentoo-kernel, it only rebuilds that one package and doesn't pull in modules. That would explain why I've only had this problem on a rebuild.

Is this how USE=dist-kernel is supposed to work?

I don't quite get the dracut-install failure mode in the kernel-rebuild case. I already installed zfs-kmod for the current kernel version, and zfs.ko is right there in file system, next to the in-tree modules that dracut-install finds.

Does dracut-install do some kind of build-ID check to reject kernel modules that don't match the exact target kernel image?

Dracut intentionally treats these problems as non-fatal (yes that is very annoying). I've been working on an upstream solution for this specifically, but we are still stuck there: https://github.com/dracut-ng/dracut-ng/pull/1215

Following, and I can weigh in on GitHub if that would be helpful. IIUC, this isn't really specific to zfs, though. I am currently installing amdgpu to my initramfs. If I wanted to do the same thing with nvidia-drivers, that would fail in the same way for the same reason. The resulting initramfs might boot, but the contents would be wrong. If that's correct, it would suggest that the fix should be general to out-of-tree modules, not just specific to zfs.

It sounds like the canonical Gentoo solution to this problem is USE=dist-kernel and @module-rebuild, and dracut upstream is the right place to talk about somehow handling this failure more coherently, at least for now.

This is how this failure is handled in ugrd, but admittedly, ugrd's ZFS support is not thoroughly tested. If you're interested in doing more testing, it would be very appreciated.
https://github.com/desultory/ugrd/commi ... 8c8ca0a764
https://github.com/desultory/ugrd/blob/ ... #L404-L411

On the note of things like GPU drivers, it depends on the overall initramfs design, but often, limiting pulled kmods/firmware to things strictly necessary to boot can increase reliability. The idea is that things such as GPU drivers can be loaded by the "real" init system, so dependencies don't have to be juggled/duplicated between the initramfs and main system.

[kwesadilo]

This is how this failure is handled in ugrd, but admittedly, ugrd's ZFS support is not thoroughly tested. If you're interested in doing more testing, it would be very appreciated.
https://github.com/desultory/ugrd/commi ... 8c8ca0a764
https://github.com/desultory/ugrd/blob/ ... #L404-L411

I take it that ugrd is essentially an alternative to dracut. At first glance, it seems like this might require a significant rearrangement of my current boot setup. I may revisit this after I get more things working on this machine, but this will probably be back-burner for now.

On the note of things like GPU drivers, it depends on the overall initramfs design, but often, limiting pulled kmods/firmware to things strictly necessary to boot can increase reliability. The idea is that things such as GPU drivers can be loaded by the "real" init system, so dependencies don't have to be juggled/duplicated between the initramfs and main system.

My original initramfs configuration was more minimal. Perhaps ironically, I added amdgpu to avoid dependency juggling. When I added consolefont to my boot services, it would sometimes fail to access the framebuffer and consequently fail to set the font. When amdgpu loads, it causes a mode set on the display device, which can take a few seconds. If consolefont starts during that mode set, it will fail. Loading the driver in the initramfs avoids this conflict. consolefont appropriately depends on modules, but the mode set takes place asynchronously after modules is supposedly fully started. I couldn't see an obvious way to express the dependency I wanted, so I just moved the module load way earlier.

[zen_desu]

This is how this failure is handled in ugrd, but admittedly, ugrd's ZFS support is not thoroughly tested. If you're interested in doing more testing, it would be very appreciated.
https://github.com/desultory/ugrd/commi ... 8c8ca0a764
https://github.com/desultory/ugrd/blob/ ... #L404-L411

I take it that ugrd is essentially an alternative to dracut. At first glance, it seems like this might require a significant rearrangement of my current boot setup. I may revisit this after I get more things working on this machine, but this will probably be back-burner for now.

On the note of things like GPU drivers, it depends on the overall initramfs design, but often, limiting pulled kmods/firmware to things strictly necessary to boot can increase reliability. The idea is that things such as GPU drivers can be loaded by the "real" init system, so dependencies don't have to be juggled/duplicated between the initramfs and main system.

My original initramfs configuration was more minimal. Perhaps ironically, I added amdgpu to avoid dependency juggling. When I added consolefont to my boot services, it would sometimes fail to access the framebuffer and consequently fail to set the font. When amdgpu loads, it causes a mode set on the display device, which can take a few seconds. If consolefont starts during that mode set, it will fail. Loading the driver in the initramfs avoids this conflict. consolefont appropriately depends on modules, but the mode set takes place asynchronously after modules is supposedly fully started. I couldn't see an obvious way to express the dependency I wanted, so I just moved the module load way earlier.

ugrd is another initramfs generator, which functions differently than dracut, and other similar udev based initramfs systems. A major design goal of ugrd is to basically work ootb, by detecting required config from the host. The main cases where user config is necessary are for advanced cryptography things which cannot be auto detected, such as the location of LUKS headers if detached, or the location of key files if used. If you're interested in testing, you'd only need to add the ugrd USE flag to installkernel. It's designed specifically to be used with installkernel and dist-kernel, but should be generally compatible with most distros. Basically, instead of trying to build an image which is generally compatible, and requires args to function at runtime, the build system tries to automatically determine what is needed based on what the current system uses, then makes an image to handle mounting that. Some options can be overridden using cmdline args, but it's rarely necessary.

As far as I know, the only ugrd module which benefits from GPU drivers is the plymouth module, but it works without them (giving a basic grey screen). I think in most cases, it's fastest and most reliable to simply do root mounting in the initramfs. ugrd's init script is also entirely serial/single threaded, with a clear goal. That means if a new device comes online during the init process, and it's not needed, it's simply ignored. It should be able to be handled later, in most cases, and attempting to handle it without considering config on the root fs (which is not yet mounted) or using config which is specified by the user at some point can make things break.

[kwesadilo]

You say you can reproduce the failure by running dracut-install args manually. In the case that dracut fails to install a kernel module, what is the exit code of the dracut process? If it is exiting with a success status code, then as far as emerge can tell, dracut succeeded. If dracut is exiting with a failure status code and the ebuild ignores that code, then that would be an ebuild problem.

Incidentally, I believe that the return code for dracut-install is effectively controlled by calling instmod -c here. I notice that add_drivers, which is used by the ZFS dracut module, passes -c, but drivers does not. However, it looks like dracut.sh just drops the exit code of instmods on the floor, which I guess is what Nowa was referring to.

[sam_]

This is further complicated by how in sys-fs/zfs, we apply 2.1.5-dracut-zfs-missing.patch which does:

Code: Select all

https://github.com/openzfs/zfs/commit/ebbfc6cb853d2d2f3f0671362d5ff5588be39e9d
https://github.com/openzfs/zfs/issues/13595
--- b/contrib/dracut/90zfs/module-setup.sh.in
+++ a/contrib/dracut/90zfs/module-setup.sh.in
@@ -19,7 +19,7 @@
 }

 installkernel() {
+       instmods zfs
-       instmods -c zfs
 }

 install() {

We do that because otherwise, you have to make sure the ZFS module is included beforehand. I guess others are less likely to see it as a result?

[Nowa]

This is further complicated by how in sys-fs/zfs, we apply 2.1.5-dracut-zfs-missing.patch which does:

Code: Select all

https://github.com/openzfs/zfs/commit/ebbfc6cb853d2d2f3f0671362d5ff5588be39e9d
https://github.com/openzfs/zfs/issues/13595
--- b/contrib/dracut/90zfs/module-setup.sh.in
+++ a/contrib/dracut/90zfs/module-setup.sh.in
@@ -19,7 +19,7 @@
 }

 installkernel() {
+       instmods zfs
-       instmods -c zfs
 }

 install() {

We do that because otherwise, you have to make sure the ZFS module is included beforehand. I guess others are less likely to see it as a result?

My PR (https://github.com/dracut-ng/dracut-ng/pull/1215) should make this workaround unnecessary. The idea is that since installkernel recognises the special exit code 77 now, we can use that to non-fatally skip all remaining hooks and effectively prevent the installation of the non-working initramfs without killing the emerge in the process.

[kwesadilo]

If you're interested in testing, you'd only need to add the ugrd USE flag to installkernel. It's designed specifically to be used with installkernel and dist-kernel, but should be generally compatible with most distros. Basically, instead of trying to build an image which is generally compatible, and requires args to function at runtime, the build system tries to automatically determine what is needed based on what the current system uses, then makes an image to handle mounting that. Some options can be overridden using cmdline args, but it's rarely necessary.

I'll keep that in mind. For now, I may have bitten off more than I can chew with Secure Boot, root on ZFS, SELinux, and a few other things I haven't tried before.

[kwesadilo]

I'm going to go ahead and call this solved. This only happens when I directly emerge gentoo-kernel as opposed to emerge @world. In that case emerge @module-rebuild afterward fixes the problem. IIUC, fixing dracut to make this more obvious requires changes that are still under discussion upstream.