Confused, UEFI / BIOS issues?

[o5gmmob8]

I am thinking that perhaps my UEFI / BIOS is buggy. I've been banging my head against the wall now for a while - this is what I did:

Code: Select all

1. rebuilt new system with 6.12.58 on external drive, snapshotted, restored to internal laptop hd as z_1000.4/gentoo.5
2. added boot entry using efibootmgr, this had the kernel name and version, 6.12.58-gentoo-dist-hardened
3. attempt to boot

I was unable to boot, it seemed like it unlocked the device, but it was unable to find the ZFS pool, I checked the name many times for typos, nada.

I attempted to use a known good initramfs (from the current running OS) and got a similar error.

I then reverted back to the original OS, but replaced the label with the OS version (I think I presently had a simpler label, Gentoo). After reverting, the same thing, it wouldn't boot, unable to find the pool.

So, I had the brilliant idea to set the label to something simple, "Gentoo 54", for 6.12.54 .... Well, that worked, I suspect either my BIOS cannot handle a long name, or hyphens or periods properly. Now what is more baffling is, the latest OS booted up even though my cmdline clearly refers to the old dataset, gentoo.4.

1. Why did my system boot to the latest dataset as the cmdline passed in the older one? I have the mountpoint set as / for the latest dataset because I was working on it, but still one would think that only matters at import time, then the next operation would be to take the dataset the user specifies and to then mount that at /, right?

2. Regarding my BIOS / UEFI, is this a known thing? I am on an HP laptop.

[NeddySeagoon]

o5gmmob8,

BIOS/UEFI are collectively known as fimware. When your CPU starts, it needs something to do.
The firmware is the something.
About 20 years ago UEFI was new, buggy and poorly tested.
Today it's BIOS (Compatibility Support Mode) that's old, buggy and badly tested.

Since you can use efibootmgr, you are in UEFI mode.

There is no such thing as a known good initrd. The initrd contains kernel modules which are tied to the kernel with what the kernel calls Version Magic. Modules that have the wrong version magic will not be loaded.
Try

Code: Select all

modinfo <some_module>

and look at the output.

As you were having problems with ZFS, I suspect the kernel module, in the initrd, wasn't being loaded.

[pingtoo]

I am thinking that perhaps my UEFI / BIOS is buggy. I've been banging my head against the wall now for a while - this is what I did:

Code: Select all

1. rebuilt new system with 6.12.58 on external drive, snapshotted, restored to internal laptop hd as z_1000.4/gentoo.5
2. added boot entry using efibootmgr, this had the kernel name and version, 6.12.58-gentoo-dist-hardened
3. attempt to boot

I was unable to boot, it seemed like it unlocked the device, but it was unable to find the ZFS pool, I checked the name many times for typos, nada.

I attempted to use a known good initramfs (from the current running OS) and got a similar error.

I then reverted back to the original OS, but replaced the label with the OS version (I think I presently had a simpler label, Gentoo). After reverting, the same thing, it wouldn't boot, unable to find the pool.

So, I had the brilliant idea to set the label to something simple, "Gentoo 54", for 6.12.54 .... Well, that worked, I suspect either my BIOS cannot handle a long name, or hyphens or periods properly. Now what is more baffling is, the latest OS booted up even though my cmdline clearly refers to the old dataset, gentoo.4.

1. Why did my system boot to the latest dataset as the cmdline passed in the older one? I have the mountpoint set as / for the latest dataset because I was working on it, but still one would think that only matters at import time, then the next operation would be to take the dataset the user specifies and to then mount that at /, right?

2. Regarding my BIOS / UEFI, is this a known thing? I am on an HP laptop.

If you got your newly created kernel come up than EFI is done, doing the right thing.

Since there are no known EFI firmware that support ZFS out of box I agree with Neddy it is something your newly created kernel missing configuration or modules.

May be do a test, use your old kernel/initramfs but with new rootfs gentoo.5? does it work?

[o5gmmob8]

Hi,

So, what I did yesterday that woked:

Code: Select all

1. old kernel / old initramfs booted to new root volume - surprisingly ZFS worked as well as wifi, but there were other errors on boot.
2. old kernel / old initramfs booted to old root volume

What didn't work:

Code: Select all

new kernel, new initramfs, new root volume

AFAIK, I rebuilt the initramfs a few times and it looked like crypt support and ZFS were included. On start, it did prompt me for a password, but it also said, something about RDSEED32 is broken, disabling the corresponding CPUID bit. After entering the LUKS passphrase, it then says failed to start ZFS Import pools by device scanning.

I think I need to look at setting rd.debug rd.shell or something so I can debug it, right now it is stopping and I cannot even poke around to help understand why.

EDIT:
I wanted to mention again that my old kernel and old initramfs will not properly boot if I use a label such as 'Gentoo 6.12.54-dist-hardened'. I'm not quite sure why, but it prompts for a passphrase, then errors out with probing the device by UUID.
I tried booting with rd.shell and rd.debug, but even with slow motion video capture on my phone, I haven't had luck yet capturing the useful bit of the console output. rd.shell doesn't seem to do anything.

zpool get -No value feature@encryption - I think that just means I didn't set encryption at the pool level which is accurate, I have LUKS on the device, once that is unlocked, then ZFS on that. I don't think that is particularly helpful, I don't think that is the culprit.

[pingtoo]

o5gmmob8,

This seems to be something I can work with. However I don't know what is the cause so will need step by step debug this could take long times with remote support.

I suggest you kept the old kernel/initrd for debug purpose for now.

In the mean time lets way a little bit, maybe someone else have better insight know where to fix.

[pingtoo]

o5gmmob8,

This seems to be something I can work with. However I don't know what is the cause so will need step by step debug this could take long times with remote support.

I suggest you kept the old kernel/initrd for debug purpose for now.

In the mean time lets way a little bit, maybe someone else have better insight know where to fix.

Well looks like no one else suggested anything yet.

If you still in the same situation and need a second pair of eyes to spot something let me know.

I will need to know how you build your initramfs, roll your own? dracut? ugrd? something else?

"rd.shell" and "rd.debug" is not kernel command line parameters so depend on your initramfs, they may or may not work.

[o5gmmob8]

Ok, that explains why rd.shell and rd.debug had no effect.

I used dracut, specifically, emerge --config gentoo-kernel.

I'm rebuilding my system again from the ground up and will see what happens this iteration. AFAIK, the logs for the boot loader bit looked clean. I'll post back once the rebuild is done.

[NeddySeagoon]

o5gmmob8,

rd.shell and rd.debug can be placed on the kernel command line but they have no meaning to the kernel.
The kernel passes them on to userspace, where it can read and parse /proc/cmdline.
User space can do whatever it wants.

The rd. prefix is used by one of the initrd generators to identify pararameters that the initrd should act on.
The rd. namespace avoids name clashes with the kernel.
The kernel knows about debug but not rd.debug. The two do quite differerent things.

Different initrd generators use different prefixes like this, so what works for one will not for another.

[pingtoo]

Ok, that explains why rd.shell and rd.debug had no effect.

I used dracut, specifically, emerge --config gentoo-kernel.

I'm rebuilding my system again from the ground up and will see what happens this iteration. AFAIK, the logs for the boot loader bit looked clean. I'll post back once the rebuild is done.

Since you are using dracut but rd.shell and rd.debug not drop you into shell?

when I say ""rd.shell" and "rd.debug" is not kernel command line parameters" I mean Linux kernel does not use them. they can still be placed on boot command line, only the user space program will interpreting them. meaning the dracut's /init should still do something with them.

if you have successful drop into initramfs's shell with dracut grenerated initramfs, there should be a log file in /run/initramfs/rdsosreport.txt. see if you can share it.

[o5gmmob8]

Ok, yeah, that makes sense, but I'm not sure why I'm not getting a recovery shell. I will see what I can find, yes, if I can extract that, that will be gold.

[NeddySeagoon]

o5gmmob8,

Is an initd being loaded?

[o5gmmob8]

Yes, but it fails. My latest rebuild is nearly done, I will reboot and try that shortly.

[pingtoo]

Yes, but it fails. My latest rebuild is nearly done, I will reboot and try that shortly.

It is hard to imagine "fail" mean here.

• Is it fail to load by EFI firmware?
• Is it fail to start by kernel?
• Is it fail because initramfs' /init not able to start?
• Is it fail for not able to find rootfs?

[o5gmmob8]

Sorry for not being clear.

It failed to find the rootfs, it found the device by UUID as it promprted me to unlock it, but it did not find the ZFS pool though it was typed properly. Switching to another init and kernel fixed that issue.

The install finished, I need to copy it over to my laptop drive and then see what happens. I used an external HDD to minimize wear and tear to my NVMe drive.

[pingtoo]

Sorry for not being clear.

It failed to find the rootfs, it found the device by UUID as it promprted me to unlock it, but it did not find the ZFS pool though it was typed properly. Switching to another init and kernel fixed that issue.

The install finished, I need to copy it over to my laptop drive and then see what happens. I used an external HDD to minimize wear and tear to my NVMe drive.

No worries, all is good. Just want to find out if something I can help.

Glad to hear it is all work out. Thanks for the information.

[o5gmmob8]

So, with the new image (ZFS dataset) along with the newly built kernel and initramfs, I get the same thing. For the record, it is still 6.12.58-gentoo-dist-hardened.

I took an image with my phone, but here is the text:

Starting Import ZFS pools by device scanning...
Failed to start Import ZFS pools by device scanning.

Failed to start systemd-cryptsetup@luks\DEVICE_UUID

[NeddySeagoon]

o5gmmob8,

After that failure, it should have dropped you to a rescue shell, or offered the option to choose a rescue shell.
Did it?

If you go into the rescue shell, can you unlock the encrypted device by issuing commands at the console?

[o5gmmob8]

No, I would expect it to, but it isn't. I must be missing a config option or something.

[o5gmmob8]

Reached target ZFS pool import target.
dracut-initqueue: Failed to start systemd-cryptsetup@luks-<UUID>.service Unit systemd-cryptsetup@luks-<UUID>.service not found

I get nothing after that, no option to get a shell, the only thing that works is CTRL+ALT+DELETE. I tried CTRL+C, CTRL+D.

My kernel cmdline:

rd.luks.uuid=<UUID> consoleblank=300 root=zfs:z_1000.4/gentoo.6 rd-info rd.break rd.shell systemd.unit=emergency.target

I will try generating the initramfs again and reading through the log more carefully.

EDIT:
I didn't notice this before, but I don't believe I have this configuration anywhere, but somehow dracut is running with --no-uefi. That seems odd to me I think if it is built without uefi, then it probably won't get whatever information is set in UEFI and therefore won't boot.

[pingtoo]

maybe try to set

Code: Select all

rd.break=pre-mount console=tty0

So this will break before zfs being import. And remove "systemd.unit=emergency.target" from boot command line, this may cause problem.

Does your dracut configuration include

Code: Select all

add_dracutmodules+=" crypt dm zfs"
# Optional force install in case dracut somehow miss needed binaries
install_items+=" /usr/lib/systemd/systemd-cryptsetup /usr/bin/cryptsetup /etc/crypttab "

Do you have /etc/crypttab on you host for dracut to copy?

Code: Select all

# <name>       <device>            <password>  <options>
luks-<UUID>    UUID=<UUID>         none        luks,discard,initramfs

Once you are in dracut emergency shell try

Code: Select all

# Optionally you can try
#
# blkid -t TYPE=crypto_LUKS
#  to search which device got the right UUID
#  and use command
# export UUID=$(blkid -s UUID -o value /dev/path/to/the/right/device/from/above
#  
#  now you can replace the "<UUID>" with $UUID
cryptsetup open /dev/disk/by-uuid/<UUID> luks-<UUID>

zpool import -f z_1000.4

exit

[o5gmmob8]

Hmm, rd.break=pre-mount console=tty0 didn't have any effect, it took about 3m for the timeout. It is checking

Code: Select all

-e /dev/disk/by-id/dm-uuid-CRYPT-LUKS?-*<UUID>*-*

Code: Select all

Entering emergency mode.  Exit the shell to continue
...
Enter root password for system maintenance
(or press Control-D to continue):

This is where I was before, no matter what I enter, my password isn't taken. I tried my LUKS passphrase, "root", "toor", nothing.

[pingtoo]

try

Code: Select all

rdinit=/bin/sh

look around, what is /init? is it systemd? or it is shell script?
do you have cryptsetup command? how about zpool command? check if all the necessary exist.

another try add SYSTEMD_SULOGIN_FORCE=1 to your kernel command line, so you can get a passwordless root shell.

finally, are you sure your keyboard is working have you tried in bootloader with keyboard array keys just to ensure keyboard is working as expect.

[o5gmmob8]

Cool, some progress. I believe rdinit=/bin/sh did the trick.

1. /init is a binary file, presumably systemd
2. hmm, I do not see a cryptsetup which is odd because it was prompting me for a password and it looked like it was cryptsetup doing the prompting - I will reboot, chroot and double-check my use flags.
3. exec /init, causes it to prompt me for my LUKS passphrase

EDIT:
1. I have cryptsetup enabled globally

[pingtoo]

Cool, some progress. I believe rdinit=/bin/sh did the trick.

1. /init is a binary file, presumably systemd
2. hmm, I do not see a cryptsetup which is odd because it was prompting me for a password and it looked like it was cryptsetup doing the prompting - I will reboot, chroot and double-check my use flags.
3. exec /init, causes it to prompt me for my LUKS passphrase

EDIT:
1. I have cryptsetup enabled globally

Not sure if you have

Code: Select all

add_dracutmodules+=" ... crypt ..."

in your dracut configuration file. It should bring in the cryptsetup binary.

And did you remove "systemd.unit=emergency.target" from your boot command line argument list?

[o5gmmob8]

1. Yes, I have crypt and zfs both there:

Code: Select all

cat /mnt/gentoo/etc/dracut.conf.d/modules.conf
    add_dracutmodules+="  crypt  zfs "

2. Yes:

Code: Select all

Boot0000* Gentoo 58     HD(1,GPT,33c89f09-e09d-45e3-a257-dbf9cacda93c,0x800,0x200000)/\EFI\gentoo\6.12.58-gentoo-dist-hardened\kernelinitrd=\EFI\gentoo\6.12.58-gentoo-dist-hardened\initramfs consoleblank=300 rd.luks.uuid=8b4c8a09-2689-4c22-bad1-dbd1e2176d43 quiet root=zfs:z_1000.4/gentoo.6 rd.info rd.break=pre-mount console=tty0 rdinit=/bin/sh SYSTEMD_SULOGIN_FORCE=1

It is strange, but I think that systemd must have that baked in, it is prompting me for a password to open that volume.