Kernel not being seen as UEFI stub (I think)

[mounty1]

I'm trying to build a Gentoo guest of a bhyve instance under OmniOS (Illumos / Solaris) but cannot boot the kernel despite following the instructions to build an EFI stub. In particular, in the config, EFI stub support is selected.

I believe that the bhyve environment implements UEFI boot correctly because it boots from the Gentoo amd64 installation CD. However, it will not boot the Gentoo kernel image.

As the bhyve config. shows, there are three disks, one each for boot/EFI, root and swap. From inside the enviroment (i.e., boot from DVD then chroot into root), they look like this:

Code: Select all

(Cheroot) livecd / # lsblk -f
NAME   FSTYPE   FSVER            LABEL                 UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  squashfs 4.0                                                                               0   100% /run/rootfsbase
sr0    iso9660  Joliet Extension Gentoo-amd64-20250216 2025-02-16-17-56-25-00                     0   100% /run/initramfs/live
vda                                                                                                        
└─vda1 vfat     FAT32            Boot                  A4FC-8281                               8.9M    50% /boot
vdb                                                                                                        
└─vdb1 ext4     1.0              Root                  d5d0fe72-73a5-4b6f-ab98-236b80b4098b   17.5G    23% /
vdc    swap     1                swap1                 c8e1c86a-dd29-4817-9dfe-a87aca546caf

then the built kernel is added to the EFI boot manager list thus:

Code: Select all

efibootmgr -c -d /dev/vda -p 1 -L "Gentoo 6.1.1 EFI Bhyve" -l '\EFI\BOOT\bootx64.efi' -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b'

and the boot manager sequence/list is:

Code: Select all

# efibootmgr
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0008,0002,0001,0000,0003,0004,0005,0006,0007
Boot0000* UiApp	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI BHYVE SATA DVD ROM BHYVE-C19E-208D-A891	PciRoot(0x0)/Pci(0x3,0x0)/Sata(0,65535,0){auto_created_boot_option}
Boot0002* UEFI Misc Device	PciRoot(0x0)/Pci(0x4,0x0){auto_created_boot_option}
Boot0003* UEFI Misc Device 2	PciRoot(0x0)/Pci(0x5,0x0){auto_created_boot_option}
Boot0004* UEFI Misc Device 3	PciRoot(0x0)/Pci(0x5,0x1){auto_created_boot_option}
Boot0005* UEFI PXEv4 (MAC:0208202FF8E9)	PciRoot(0x0)/Pci(0x6,0x0)/MAC(0208202ff8e9,1)/IPv4(0.0.0.00.0.0.0,0,0){auto_created_boot_option}
Boot0006* UEFI HTTPv4 (MAC:0208202FF8E9)	PciRoot(0x0)/Pci(0x6,0x0)/MAC(0208202ff8e9,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri(){auto_created_boot_option}
Boot0007* EFI Internal Shell	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
Boot0008* Gentoo 6.1.1 EFI	HD(1,GPT,caf2dfc5-7494-43a5-99d7-1c412719fc2d,0x800,0x9000)/File(\EFI\Gentoo\linuz-6.1.1.efi)72006f006f0074003d0055005500490044003d00640035006400300066006500370032002d0037003300610035002d0034006200360066002d0061006200390038002d00320033003600620038003000620034003000390038006200

so the built kernel (0008) should be booted in preference to the DVD (0001) but it's ignored/skipped-over (as opposed to found, booted and the boot fails). If I remove the DVD from the bhyve config., the machine just tries and fails to boot via PXE.

Again I believe I've set up the ESP partition correctly. It was formatted with the command:

Code: Select all

mkfs.fat -F 32 -n Boot /dev/vda1

and contains:

Code: Select all

# ls -lR /boot/
/boot/:
total 1711
drwxr-xr-x 4 root root     512 Mar 30 05:40 EFI
-rwxr-xr-x 1 root root 1751347 Mar 28 03:21 System.map

/boot/EFI:
total 1
drwxr-xr-x 2 root root 512 Mar 30 05:41 BOOT
drwxr-xr-x 2 root root 512 Mar 29 07:01 Gentoo

/boot/EFI/BOOT:
total 3638
-rwxr-xr-x 1 root root 3724352 Mar 30 05:41 bootx64.efi

/boot/EFI/Gentoo:
total 3638
-rwxr-xr-x 1 root root 3724352 Mar 28 03:21 linuz-6.1.1.efi

where linuz-6.1.1.efi and bootx64.efi are copies of /usr/src/linux/arch/x86/boot/bzImage. The partition is correctly configured for ESP:

Code: Select all

# parted /dev/vda print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 21.0MB
Sector size (logical/physical): 512B/8192B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  19.9MB  18.9MB  fat32        EFI system partition  boot, esp

I really cannot see what I've missed. What else does the setup need to make it bootable?

Can I ask that if you want to contribute a peripheral/irrelevant comment such as the kernel is quite old, or it's ESP partition so why is it mounted at /boot etc., that you wait for a real answer so that the question, which has taken me over an hour to put together, is seen as not-replied-to by someone who might actually know the answer. Thank you.

[pietinger]

[...] then the built kernel is added to the EFI boot manager list thus:

Code: Select all

efibootmgr -c -d /dev/vda -p 1 -L "Gentoo 6.1.1 EFI Bhyve" -l '\EFI\BOOT\bootx64.efi' -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b'

Do you have a kernel WITH an initramfs or WITHOUT? I ask because the kernel does not understand "root=UUID=... ".
(see more: https://wiki.gentoo.org/wiki/User:Pieti ... %3DUUID%3D )

IF you have a kernel WITH an initramfs THEN you are missing the parameter "initrd=".
(sse more: https://wiki.gentoo.org/wiki/User:Pieti ... _initrd.3D )

BTW: You should not use EFI\BOOT\bootx64.efi because this is a very special name (every UEFI must recognize this WITHOUT having a UEFI-entry); use a normal name.
=>

Code: Select all

efibootmgr -c -d /dev/vda -L "Gentoo 6.1.1 EFI Bhyve" -l '\EFI\BOOT\mykernel.efi' -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b initrd=\EFI\BOOT\NAMEOFINITRDIMAGE'

- OR -

Code: Select all

efibootmgr -c -d /dev/vda -L "Gentoo 6.1.1 EFI Bhyve" -l '\EFI\BOOT\mykernel.efi' -u 'root=PARTUUID=.........'

[NeddySeagoon]

mounty1,

Please share your kernel .config on a pastebin.
With a missing initrd, you should at least get a kernel panic message. That you don't suggests it a console configuration error.

... a Gentoo guest of a bhyve instance ...

How do you get the guest console and at what point in the boot process?

[mounty1]

Please share your kernel .config on a pastebin.
With a missing initrd, you should at least get a kernel panic message. That you don't suggests it a console configuration error.

I did. No initrd but as the VM does not find or attempt to boot the kernel, this is irrelevant.

How do you get the guest console and at what point in the boot process?

bhyve provides access to the console via VNC thus, available as soon as the VM is started. That's how I know that the VM does not attempt to boot the Gentoo kernel.

[mounty1]

Do you have a kernel WITH an initramfs or WITHOUT? I ask because the kernel does not understand "root=UUID=... ".

without, but as the VM does not find or attempt to boot the Gentoo kernel, this is irrelevant. Obtaining a kernel panic would be progress.

[NeddySeagoon]

mounty1,

Does EFI/Gentoo/linuz-6.1.1.efi exist in your ESP?

Found your .config. I missed it last time.

Code: Select all

# CONFIG_DRM is not set
 
#
# ARM devices
#
# end of ARM devices
 
#
# Frame buffer Devices
#
# CONFIG_FB is not set
# end of Frame buffer Devices

You have no video drivers at all configured.

It looks like you have the console mapped to a real serial port

Code: Select all

CONFIG_SERIAL_8250_CONSOLE=y

You need to set the port and configuration on the kernel command line to do that.
Then /etc/inittab needs to be changed to start agtty on that port, with the same configuration, or you won't get a login prompt.

[pietinger]

without, but as the VM does not find or attempt to boot the Gentoo kernel, this is irrelevant. Obtaining a kernel panic would be progress.

Hmm ... what do you mean with "VM does not find the kernel"? Maybe the VM loads the kernel and you just dont see anything because of a missing console? Maybe you have a invisible kernel panic?

If you have solved the output you should also check these options - because you will need one of them that kernel is able to access the virtual device:

Code: Select all

# CONFIG_BLK_DEV_NVME is not set
# CONFIG_SCSI_VIRTIO is not set
# CONFIG_ATA is not set

Maybe this could improve the performance:

Code: Select all

# CONFIG_IOMMU_SUPPORT is not set

BTW: Kernel 6.1.1 is really very outdated

[pingtoo]

mounty1,

May be you can try to boot in to "Boot0007* EFI Internal Shell" see if you can see your device(s).

In efi shell you can use command "devices" to list all devices EFI drivers managed.

[mounty1]

Hmm ... what do you mean with "VM does not find the kernel"? Maybe the VM loads the kernel and you just dont see anything because of a missing console? Maybe you have a invisible kernel panic?

By the time I can open the console (i.e., start the VNC viewer) the VM (bhyve supervisor) is trying to boot via PXE.

I'll write it again: the VM is trying to boot via PXE. It has skipped-over the Gentoo boot entry. Unless the UEFI system has tried to boot the Gentoo kernel and somehow worked out that it has a faulty configuration, it has skipped over it or put another way, has not attempted to boot it.

[mounty1]

Does EFI/Gentoo/linuz-6.1.1.efi exist in your ESP?

I already listed (ls -lR) the entire ESP, including EFI/Gentoo/linuz-6.1.1.efi so yes.

[NeddySeagoon]

mounty1,

Code: Select all

linuz-...

I'm more used to bzImage
Is an odd (to me) filename for a kernel. What does file say about it?
I'm more used to bzImage.

If the firmware choked on your kernel file, it would indeed move on.

[pingtoo]

Hmm ... what do you mean with "VM does not find the kernel"? Maybe the VM loads the kernel and you just dont see anything because of a missing console? Maybe you have a invisible kernel panic?

By the time I can open the console (i.e., start the VNC viewer) the VM (bhyve supervisor) is trying to boot via PXE.

I'll write it again: the VM is trying to boot via PXE. It has skipped-over the Gentoo boot entry. Unless the UEFI system has tried to boot the Gentoo kernel and somehow worked out that it has a faulty configuration, it has skipped over it or put another way, has not attempted to boot it.

Does this(boot PXE) mean you don't have a way to choose which entry to boot?

EFI will skip/ignore entries when it is not available, so one way to understand would be go into EFI shell to see how the pre-boot firmware see the environment.

[mounty1]

Code: Select all

# file /mnt/boot/EFI/Gentoo/linuz-6.1.1.efi 
/mnt/boot/EFI/Gentoo/linuz-6.1.1.efi: Linux kernel x86 boot executable bzImage, version 6.1.1-gentoo (root@livecd) #2 PREEMPT_DYNAMIC Fri Mar 28 03:18:55 -00 2025, RO-rootFS, swap_dev 0X3, Normal VGA

[mounty1]

Does this(boot PXE) mean you don't have a way to choose which entry to boot?

Yes, it does. I do not have a choice, such as a menu of possible boot options. Normally I'd expect to have to press a key (e.g., on my Thinkpad laptops it's F12) to get a menu.

EFI will skip/ignore entries when it is not available, so one way to understand would be go into EFI shell to see how the pre-boot firmware see the environment.

Thanks; I'll try that.

[NeddySeagoon]

mounty1,

My EFI stub kernel, which i did exactly once to bring this machine up says

Code: Select all

# file /boot/EFI/EFI/Boot/bootx64.efi 
/boot/EFI/EFI/Boot/bootx64.efi: Linux kernel x86 boot executable, bzImage, version 5.14.1-gentoo (root@neddy_static) #4 SMP PREEMPT Sat Sep 11 13:54:52 BST 2021, RO-rootFS, Normal VGA, setup size 512*27, syssize 0x5d12a, jump 0x26c 0x8cd88ec0fc8cd239 instruction, protocol 2.15, from protected-mode code at offset 0x3a8 0x5c6869 bytes LZMA compressed, relocatable, handover offset 0x190, legacy 64-bit entry point, can be above 4G, 64-bit EFI handoff entry point, xloadflags bit 5, max cmdline size 2047, init_size 0x1d89000

I don't know if the rest of the waffle that I have but you don't, matters.
In the interests of full disclosure, my kernel is a mixed mode kernel.

[mounty1]

Code: Select all

64-bit EFI handoff entry point

That has to be significant. Why does my kernel not have that? I ask myself.

[Later] tried a make clean && make -s but my file output is unchanged.

[mounty1]

Code: Select all

BdsDxe: failed to load Boot0008 "Gentoo 6.1.1 EFI" from HD(1,GPT,CAF2DFC5-7494-43A5-99D7-1C412719FC2D,0x800,0x9000)/\EFI\Gentoo\linuz-6.1.1.efi: Not found
BdsDxe: failed to load Boot0002 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0): Not Found
BdsDxe: failed to load Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x5,0x0): Not Found
BdsDxe: failed to load Boot0002 "UEFI Misc Device 3" from PciRoot(0x0)/Pci(0x5,0x1): Not Found

>>Start PXE over IPv4

I wonder if the failure to find the kernel is related to my removing the CD-ROM from the VM configuration.

Eventually it gets to the UEFI shell and the map command includes in its output:

Code: Select all

BLK1: Alias(s):
      PciRoot(0x0)/Pci(0x4,0x0)/HD(1,GPT,CAF2DFC5-7494-43A5-99D7-1C412719FC2D,0x800,0x9000)

so I'm a bit surprised by this command and output:

Code: Select all

Shell> ls blk1:
Directory of: blk1:
          0 File(s)               0 bytes
          0 Dir(s)
Shell>

It seems that for some reason my ESP is not readable. The creation command was in the OP.

[pingtoo]

Do you know which device is "CAF2DFC5-7494-43A5-99D7-1C412719FC2D" is ? It is not show in your OP.

The message

BdsDxe: failed to load Boot0008 "Gentoo 6.1.1 EFI" from HD(1,GPT,CAF2DFC5-7494-43A5-99D7-1C412719FC2D,0x800,0x9000)/\EFI\Gentoo\linuz-6.1.1.efi: Not found

indicate your have a storage that that is labeled by "GPT" and the first partition "HD(1,...) expect to have filesystem UUID of "CAF2DFC5-7494-43A5-99D7-1C412719FC2D"

What if you do "map -r", do you get more aliases? If not you will need to work out from the bhyve level. because you at least need to make EFI see the vda device.

I think you need to make you efi see "A4FC-8281"/

does your "map" command have more output?

-- edit --
Does pressing "F2" at the time VNC connect (or thereafter) bring up a BIOS menu?
-- edit --

[Zucca]

This is just a heads up, may not be the case at all here but:

• I had similar experiences on my Dell Precision 7810.
  I couldn't get rEFInd to load (which also required kernel to have EFI_STUB). And I think I also tried to boot straight to kernel too.
  I think it has a somewhat broken UEFI implementaion.
  
  I finally got the kernel to boot by (basically giving up and) installing grub (which is also a boot loader in addition to boot manager, while rEFInd is only latter and requires EFI_STUB kernel) and passing --removable to grub-install.
  The --removable option places grub EFI binary into ${ESP}\EFI\BOOT\BOOTX64.EFI, aka. "the default location" where all UEFI implementations should find it. I still don't have clue which part failed: kernel EFI_STUB loading or other boot managers (rEFInd).
  So in case you get as desperate as I did - you could try my method. ;)

[mounty1]

Do you know which device is "CAF2DFC5-7494-43A5-99D7-1C412719FC2D" is ? It is not show in your OP.

ESP partition UUID

Code: Select all

(Cheroot) livecd / # blkid
/dev/sr0: BLOCK_SIZE="2048" UUID="2025-02-16-17-56-25-00" LABEL="Gentoo-amd64-20250216" TYPE="iso9660" PTTYPE="PMBR"
/dev/vdb1: LABEL="Root" UUID="d5d0fe72-73a5-4b6f-ab98-236b80b4098b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="3391b633-01"
/dev/loop0: BLOCK_SIZE="131072" TYPE="squashfs"
/dev/vdc: LABEL="swap1" UUID="c8e1c86a-dd29-4817-9dfe-a87aca546caf" TYPE="swap"
/dev/vda1: LABEL_FATBOOT="Boot" LABEL="Boot" UUID="A4FC-8281" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="caf2dfc5-7494-43a5-99d7-1c412719fc2d"

[pingtoo]

I recently learned that EFI actually expect use filesystem's UUID.(Previous statement is incorrect, please see post=8859049) That being said, I see that in EFI shell you got a BLK1 device that point to this device with empty content. So I am not 100% sure how this happen.

So few things to try, modify or add new entry with "A4FC-8281" to see if this help EFI.(No need to do this as it is based on mistake about)

Did you tried with my other suggestions? in EFI Shell, try "map" command and see more aliases? I suspect you should at lease see one more BLK0 but I am hoping you also see FS0 or more.

And does "F2" work in VNC?

If none of above give some thing for further actions, then we will need to try at zonecfg level. I don't know bhyve and last time I do zonecfg may be 10 years ago on Solaris 11. so we will need some try and error to figure this out.

[mounty1]

F2 does not work with bhyve. It's straight into UEFI.

I haven't tried map because it's pretty obvious what the problem is. The command

Code: Select all

efibootmgr -c -d /dev/vda -p 1 -L "Gentoo 6.1.1 EFI" -l '\EFI\Gentoo\linuz-6.1.1.efi' -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b'

inserts the partition UUID into the UEFI entry, but UEFI wants the filesystem UUID. So I must either force efibootmgr to insert the filesystem UUID (can't see any option for that) or make the filesystem UUID the same as the partition UUID (they're of very different lengths).

zonecfg doesn't really help, as far as I can tell, as it doesn't provide any option for tweaking the UEFI boot entry or sequence. Really I need some way to make efibootmgr insert the filesystem UUID, or to adjust the Gentoo entry.

[pingtoo]

F2 does not work with bhyve. It's straight into UEFI.

I haven't tried map because it's pretty obvious what the problem is. The command

Code: Select all

efibootmgr -c -d /dev/vda -p 1 -L "Gentoo 6.1.1 EFI" -l '\EFI\Gentoo\linuz-6.1.1.efi' -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b'

inserts the partition UUID into the UEFI entry, but UEFI wants the filesystem UUID. So I must either force efibootmgr to insert the filesystem UUID (can't see any option for that) or make the filesystem UUID the same as the partition UUID (they're of very different lengths).

zonecfg doesn't really help, as far as I can tell, as it doesn't provide any option for tweaking the UEFI boot entry or sequence. Really I need some way to make efibootmgr insert the filesystem UUID, or to adjust the Gentoo entry.

Your post make me think you did not type in the "efibootmgr -c -d /dev/vda -p 1 -L "Gentoo ...", am I understand correct? if so what generate that command?

Working from EFI really does not necessary solve the problem, it is just a step that try to verify where action(s) really need to occur. However if we can identify in EFI Shell that boot device/filesystem actually is not accessible by EFI Shell than we know issue is at host level. Using "map" command can give us cue on what EFI system see (Since you don't have a way to access firmware boot menu) so please give it a try when you can.

[sMueggli]

I recently learned that EFI actually expect use filesystem's UUID.

Where?

[pingtoo]

sMueggli,

Thank you for reminding me about this.

My statement was incorrect. It is indeed should be Partition UUID.

so in essence the UUID in that "efibootmgr ... -u 'root=UUID=d5d0fe72-73a5-4b6f-ab98-236b80b4098b'" the UUID value is correctly identify a partition.

My reference about the "learning" part was from this post.

The reason my statement about "FS UUID" is because I under impression then I made a mistake about what should be value of UUID but did not realize that mistake then was comparing DISK UUID vs PARTITION UUID

I should make a amendment following post about "correct UUID value in this thread"