Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ZFS Gentoo failed to boot initramfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Tue Mar 26, 2024 10:44 pm    Post subject: ZFS Gentoo failed to boot initramfs Reply with quote

I am using zfsbootmenu with gentoo-sources, and for some reason I would boot of my kernel, it will try to load the initramfs and will shoot me back to a screen where I can press F9 to manually boot again. I tried downgrading zfs utils to 2.1.15, I added the kexec-tools patch. I tried using 6.1 kernel version and still the same issue. I am now using 6.6.21, but for some reason 6.6.13 works, but I haven't reconfigured it yet due to the fact 6.6.13 is now masked for amd64. I am pretty desperate that I don't know if installing zfs statically would work or rather if static is the best choice to go with. I don't know what to post here so to say that could help with the issue besides my .config file and the packages I have installed on my machine.

CONFIG LINK:
https://termbin.com/41lv

I also noticed that docker's overlayfs can be an issue if you're running zfs... Well at least in my experience.

equery list '*':

https://termbin.com/smlw

The packages above are installed on my system, which also probably need to be cleaned out as I swapped over to wayland + sway with ryzen cpu. For some reason my system has used 64GB of space which literally makes no sense at all. So even finding packages that are needed will be so helpful for me
Back to top
View user's profile Send private message
turtles
Veteran
Veteran


Joined: 31 Dec 2004
Posts: 1656

PostPosted: Thu Mar 28, 2024 9:18 pm    Post subject: Re: ZFS Gentoo failed to boot initramfs Reply with quote

kucklehead wrote:
I am using zfsbootmenu with gentoo-sources, and for some reason I would boot of my kernel, it will try to load the initramfs and will shoot me back to a screen where I can press F9 to manually boot again....
I am now using 6.6.21, but for some reason 6.6.13 works, but I haven't reconfigured it yet due to the fact 6.6.13 is now masked for amd64
.


Can you diff your 6.6.13 and 6.6.21 kernel configs and see what the differences are?
_________________
Donate to Gentoo
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Tue Apr 02, 2024 3:42 pm    Post subject: Reply with quote

Here is the requested info:
https://bpa.st/PY6Q

There should be a lot more configurations outputted instead of three or four as I recently configured 6.6.21 to support docker, but I accidentally overwritten the config file with the bzImage, so I just reused my 6.6.13 config
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Tue Apr 02, 2024 8:42 pm    Post subject: Re: ZFS Gentoo failed to boot initramfs Reply with quote

kucklehead wrote:
I am using zfsbootmenu with gentoo-sources, and for some reason I would boot of my kernel, it will try to load the initramfs and will shoot me back to a screen

Using "zfsbootmenu with gentoo-sources" does not lead a clear picture of the newly built kernel can boot with zfs. So question,

After you build your new kernel did you re-merge zfs-kmod?

I don't use zfs so this could be a dumb question, Do you also need to re-build initrd after you got a new zfs ready kernel?

These are just a few things to check for boot failure.
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Wed Apr 03, 2024 1:19 am    Post subject: Reply with quote

I did in fact re-emerge zfs-kmod, right before I create/rebuild the initramfs/initrd with dracut. And of course this is after I compile the modules and install them. I noticed the configurations are slightly different, which you can view from my previous post. I am going to try to add the configurations to 6.6.21, and configure docker once again, and see if that was the issue all along.
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Wed Apr 03, 2024 1:20 am    Post subject: Reply with quote

I did in fact re-emerge zfs-kmod, right before I create/rebuild the initramfs/initrd with dracut. And of course this is after I compile the modules and install them. I noticed the configurations are slightly different, which you can view from my previous post. I am going to try to add the configurations to 6.6.21, and configure docker once again, and see if that was the issue all along.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Wed Apr 03, 2024 3:22 pm    Post subject: Reply with quote

kucklehead,

Since you did everything right, I am not sure what else at this time.

However if you wish to continue work with me in this conversation, I will be happy to serve as second pair of eye to go through the setup process, I plan to review the zfsbootmenu document and gentoo zfs document and ask series of questions. this is in hope that in my questions that will ring a bell for you to see if there is any step(s) you might have missed during the rebuild process.

There is always a chance the new kernel may have something incompatible with your setup, however I believe it is easier to review existing processes to verify that all steps were correctly executed before dig in to code to see where the compatibility issue rise.
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Thu Apr 04, 2024 5:15 pm    Post subject: Reply with quote

That would be really helpful if you could serve as a second pair of eyes; however, my responses will be random and not consistent due to the fact that I am still an undergrad. I do have notes that go through all the steps of installing zfs on Linux. I already have two successful zfs installations, which are Ubuntu and Arch Linux. My Ubuntu is using Grub, and my Arch Linux is using Bootctl. I think that is the name. 

The user manual for zfsbootmenu kind of confused me at first when it talked about the boot environment. I was thinking the boot environment was the kernel itself, so I pointed everything to my boot partition. But my logic was flawed because the boot environment is actually the system itself, so whatever dataset has a root, make sure it points to it. I do not want you to make the same mistake I did. 

That said, there was an issue with a certain version of zfsbootmenu. I believe 2.0 wouldn't detect the scripts I created. Normally, you could store the scripts inside /boot/efi/hooks (the folder is probably not hooks; it is just an example).  These bash scripts I made would unlock my Linux partition and so forth. To fix this issue, I had to upgrade to 3.0. This information is probably vague or partially true because someone explained this to me in the zfsbootmenu channel or zfs channel roughly a month ago, but it should give an idea of how difficult zfsbootmenu can be at times. 

Perhaps you already know this; however, I am saying this for those who are interested in zfsbootmenu and do not know as much about it. The zfs-gentoo wiki is still under construction the last time I looked, and it does not have sections that walk you through zfs-gentoo-sources. This may be vague or not clear, so to sum it up, they have a section where they download an EFI and use kernel-bin; this section also assumes root is not inside the Linux partition. They do not have a section where it shows the user how to generate an EFI file using generate-zbm and does not show steps to help users who have root inside the Linux partition. I was planning on cleaning up my notes and seeing if I could fill those sections up. They do need to be cleaned up and reworded to avoid copyright infringement.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Thu Apr 04, 2024 9:38 pm    Post subject: Reply with quote

kucklehead,

I am glad you accept my proposal. I hope we will be able to find where the culprit and learn something new.

No worries about times. I am retired I can always find something else to do in the mean time.

I think we need to sync our terminology.

I am not sure when you refer "boot partition" what does it mean in the ZFS world. Do you mean a zvol? And from your initial post, you said "shoot me back to a screen where I can press F9", I haven't been using PC for many years now. so I am not sure what does it mean, my guess it take you back to BIOS and let you select where to boot from?

Are your FBL (First Boot Loader) is BIOS (old PC way) or EFI?
Back to top
View user's profile Send private message
turtles
Veteran
Veteran


Joined: 31 Dec 2004
Posts: 1656

PostPosted: Fri Apr 05, 2024 4:10 am    Post subject: Reply with quote

kucklehead wrote:
Here is the requested info:
https://bpa.st/PY6Q

There should be a lot more configurations outputted instead of three or four as I recently configured 6.6.21 to support docker, but I accidentally overwritten the config file with the bzImage, so I just reused my 6.6.13 config


Thanks, noting jumps out at me in that diff its pretty small:
Code:
# diff /boot/config-6.6.21-gentoo-x86_64 /boot/config-6-6-13-gentoo-x86_64
3c3
< # Linux/x86 6.6.21-gentoo Kernel Configuration
---
> # Linux/x86 6.6.13-gentoo Kernel Configuration
18d17
< CONFIG_GCC_ASM_GOTO_OUTPUT_WORKAROUND=y
4372a4372
> CONFIG_TOUCHSCREEN_TI_AM335X_TSC=m
5674a5675
> CONFIG_MFD_TI_AM335X_TSCADC=m
9609a9611
> CONFIG_TI_AM335X_ADC=m

_________________
Donate to Gentoo
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Mon Apr 15, 2024 4:57 pm    Post subject: Reply with quote

I am currently using EFI. What you said is exactly what is happening. It shoots me back the bios where I can select my vmlinuz.EFI. Anyways, I honestly thought that me building in docker support was the issue, and the possible solution to fix that would be statically linking the zfs module and build the kernel. I downgraded to zfs-2.1.15 which now supports 6.7 kernels. If I understand correctly, the zfs module will not be loaded during runtime, and therefore, if there was an issue with docker, it would have been probably resolved. The only other issue I can think of is the firmware. I am currently using amdgpu and I have the blobs built into the initramfs. There was an error I was getting in my dmesg, something like this: " amdgpu: Secure display: Generic Failure." and all I could find about that error was there was something wrong with the new firmware.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Mon Apr 15, 2024 6:51 pm    Post subject: Reply with quote

kucklehead,

Allow me to introduce another Gentoo forums thread:zfsbootmenu issues booting stucks?The thread share similar interest so it could be a good reference.

I will reiterate a point I made in above thread, "zfsbootmenu" introduce a linux kernel in its own, then user of zfsbootmenu can have another linux kernel that will be the finial runtime environment.

So can you clarify which kernel image you reference in your first post. one you used for zfsbootment or kernel to be used in final runtime.

You said you build with docker what is build with docker? the zfsbootmenu image? linux kernel?, initramfs? or something else?

Was your zfsbootmenu a "bundle"? or it is separated in a kernel + initramfs?

My questions could be bit of out of sync in a sense that is not relevant, but please forgive me, I have lost my original train of thoughts, so kind of need a refreshment.

I think the amdgpu module should not get into play inside zfsbootmenu yet, as long as the kernel inside the zfsbootmenu system inherit EFI's setting you would not need a special display driver. I mean the kernel inside the zfsbootmenu should have framebuffer to support what EFI initialised display system.
Back to top
View user's profile Send private message
kucklehead
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2020
Posts: 108

PostPosted: Thu Apr 18, 2024 4:48 pm    Post subject: Reply with quote

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PREREQS:
Make sure you have enabled the boot flag for systemd package

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step one:
Code:


ZFSBootMenu attempts to match one of several possible initramfs names for each kernel it identifies. Broadly, an initramfs is paired with a kernel when its name matches one of four forms:

initramfs-${label}${extension}

initramfs${extension}-${label}

initrd-${label}${extension}

initrd${extension}-${label}

NOTE: Step one will only work if you already compiled your kernel and copied the bzImage over to your /boot/efi folder
SUMMARY: your vmlinuz must match the kernel folder. For example, my kernel folder is 6.6.21-gentoo-x86_64. That means, I need to make sure that my vmlinuz is this: vmlinuz-6.6.21-gentoo-x86_64

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step Two:
I am using generate-zbm. If you do not know what generate-zbm is, it will create the vmlinuz.EFI for you, if and only if you configure your config.yaml correctly. This is my config.yaml:
Code:

Global:
  ManageImages: true
  BootMountPoint: /boot/efi
  DracutConfDir: /etc/zfsbootmenu/dracut.conf.d
  PreHooksDir: /etc/zfsbootmenu/generate-zbm.pre.d
  PostHooksDir: /etc/zfsbootmenu/generate-zbm.post.d
  InitCPIO: false
  InitCPIOConfig: /etc/zfsbootmenu/mkinitcpio.conf
Components:
  ImageDir: /boot/efi/EFI/zbm
  Versions: 3
  Enabled: false
  syslinux:
    Config: /boot/syslinux/syslinux.cfg
    Enabled: false
EFI:
  ImageDir: /boot/efi/EFI/zbm
  Stub: /usr/lib/systemd/boot/efi/linuxx64.efi.stub
  #Stub: /usr/lib/systemd/boot/efi/systemd-bootx64.efi
  Versions: false
  Enabled: true
Kernel:
  #CommandLine: root=ZFS=rpool/ROOT/gentoo ro quiet loglevel=0
  CommandLine: lsm=landlock,lockdown,yama,integrity,apparmor,bpf apparmor=1 security=apparmor i8042.nopnp splash nowatchdog pcie_aspm.policy=powersave amd_pstate=passive ro quiet spl.spl_hostid=0x132255fe loglevel=0
  #Prefix: /boot/vmlinuz-6-7-1-gentoo-r1-x86_64 /boot/vmlinuz-6-7-2-gentoo-r1-x86_64

NOTE: The # means it is commented out, so there shouldn't be an issue with generate-zbm creating the vmlinuz.EFI

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step Five, compiling the kernel:
I thought that creating zfs as a static package could fix the issue, but my understanding is somewhat flawed. I understand that buidling the zfs module into the kernel is literally the same as [*] when configuring your kernel.
That said, these are my steps I followed:
NOTE: I went to openzfs github and checked the latest zfs version that is supported which is 2.1.15
Code:

copied my config file from /boot/ to /usr/src/linux/.config

--enable-linux-builtin --with-linux=$LINUX_DIR --with-linux-obj=$LINUX_DIR
ln -s x86 /usr/src/linux/arch/amd64
ebuild /var/db/repos/gentoo/sys-fs/zfs/zfs-2.1.15.ebuild clean unpack
cp /var/tmp/portage/sys-fs/zfs-2.0.5/work/zfs-2.1.15/config/kernel.m4{,.orig}
nanpo -w /var/tmp/portage/sys-fs/zfs-2.0.5/work/zfs-2.1.15/config/kernel.m4
mkdir -p /etc/portage/patches/sys-fs/zfs



NOTE: your llvm patch should look something like this:
Code:

--- zfs-2.0.5/config/kernel.m4.orig     2021-09-11 21:32:30.967155385 +0000
+++ zfs-2.0.5/config/kernel.m4  2021-09-11 21:37:10.331820894 +0000
@@ -527,7 +527,7 @@
 # Example command line to manually build source
 # make modules -C $LINUX_OBJ $ARCH_UM M=$PWD/build/$1
 
-ccflags-y := -Werror $FRAME_LARGER_THAN
+ccflags-y := -Werror -Wno-address-of-packed-member $FRAME_LARGER_THAN
 _ACEOF
 
        dnl # Additional custom CFLAGS as requested.
@@ -585,7 +585,7 @@
 AC_DEFUN([ZFS_LINUX_COMPILE], [
        AC_TRY_COMMAND([
            KBUILD_MODPOST_NOFINAL="$5" KBUILD_MODPOST_WARN="$6"
-           make modules -k -j$TEST_JOBS -C $LINUX_OBJ $ARCH_UM
+           make LLVM=1 LLVM_IAS=1 modules -k -j$TEST_JOBS -C $LINUX_OBJ $ARCH_UM
            M=$PWD/$1 >$1/build.log 2>&1])
        AS_IF([AC_TRY_COMMAND([$2])], [$3], [$4])
 ])

NOTE: that means you need to use a editor tool such as nano and search for ZFS_LINUX_COMPILE and config-y and ONLY add make LLVM=1 LLVM_IAS=1 modules -k -j$TEST_JOBS -C $LINUX_OBJ $ARCH_UM into ZFS_LINUX_COMPILE

Then execute this:
Code:

(cd /var/tmp/portage/sys-fs/zfs-2.1.15/work/ && diff -u zfs-2.1.15/config/kernel.m4{.orig,} > /etc/portage/patches/sys-fs/zfs/llvm.patch)

After the patch looks similar, you need to cd into your /usr/src/linux run make menuconfig and enable ZFS and the OVERLAY SUPPORT, save it and run make prepare
Then, finally run these commands:
Code:

env EXTRA_ECONF='--enable-linux-builtin --with-linux=/usr/src/linux --with-linux-obj=/usr/src/linux' ebuild /var/db/repos/gentoo/sys-fs/zfs/zfs-2.1.15.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-2.0.5/work/zfs-2.1.15/ && ./copy-builtin /usr/src/linux)

Then run these commands:
Code:

make -j$(nproc)
make -j$(nproc) modules
make modules_install

SOURCES: https://github.com/openzfs/zfs/issues/10450#issuecomment-643654436
Step four, creating the initramfs:
NOTE: If you're using dracut which I am, you have to make sure that your .confs are setup correctly. That means in /etc/zfsbootmenu/dracut.conf.d/ I have 99-crypt.conf, omit-drivers.conf, zfsbootmenu.conf. I don't know if genkernel can be used, that is something you need to figure out.
Step Five, Finalizing:
Once you created the initramfs, now you can run generate-zbm and it will look for your vmlinuz and it should create your vmlinuz.EFI

b]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------[/b]

The steps I outlined should answer your questions and if not....
Quote:

You said you build with docker what is build with docker? the zfsbootmenu image? linux kernel?, initramfs? or something else?

yes I built docker support into my kernel by editing my config file to include [*]. So it is not built into the zfsbootmenu kernel, but is built into 6.6.21 kernel. I have zfs ubuntu that had overlayfs installed with docker, which made the system very unstable for me. So my initial thought was: Could docker and zfs not be compatiable? However, I tested this theory by building the zfs module into my kernel which the version is 6.6.21 and still the same results.
Quote:

Was your zfsbootmenu a "bundle"? or it is separated in a kernel + initramfs?

Yes, it is bundled together. I had to make sure that it was:
Quote:

If you will create unified EFI executables (which bundles the kernel, initramfs and command line), you will also need a an EFI stub loader, which is typically included with systemd-boot or gummiboot.

SOURCE: https://docs.zfsbootmenu.org/en/v2.3.x/
Quote:

I will reiterate a point I made in above thread, "zfsbootmenu" introduce a linux kernel in its own, then user of zfsbootmenu can have another linux kernel that will be the finial runtime environment.

You are correct on that. zfsbootmenu created it's own linux kernel which is a linux system that will boot another kernel which should boot the initramfs.
Quote:

I think the amdgpu module should not get into play inside zfsbootmenu yet, as long as the kernel inside the zfsbootmenu system inherit EFI's setting you would not need a special display driver. I mean the kernel inside the zfsbootmenu should have framebuffer to support what EFI initialised display system.

My only conclusion was perhaps the firmware is unstabled. But I can be wrong. I asked a similar question in zfsbootmenu channel, and the response I got was to try out 6.6.24 or 25 instead becuase 6.6.21 has some issues: https://lore.kernel.org/stable/20240402172908.4137792-1-steve.wahl@hpe.com/

Quote:

So can you clarify which kernel image you reference in your first post. one you used for zfsbootment or kernel to be used in final runtime.

The kernel I was referring to is the zfsbootmenu kernel. It fails to load my 6.6.21 kernel.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Thu Apr 18, 2024 6:58 pm    Post subject: Reply with quote

kucklehead,

Thank you very much for the detail explain, much appreciated.

Quote:
The kernel I was referring to is the zfsbootmenu kernel. It fails to load my 6.6.21 kernel.
Let's not jump into conclusion that kernel in zfsbootmenu is to blame. Through out this thread I have yet found something that indicate zfsbootmenu cause problem. Could it be possible that 6.6.21 kernel is simply not function therefor it crashed and reboot and take you to the firmware manual? Have you done something to rule out this possibility?

Now, go to the assume something wrong with zfsbootmenu route.

Question: module complied with wrong compiler?
In your step five compiling the kernel, The patch code I think it is for zfs module build, not necessary for kernel build, So I need to ask, Is your kernel in zfsbootment also build with LLVM? As far as I know linux kernel is picky about how its module build, if different version of compiler were used, something it will not accept(load) the module at runtime.

Testing,
You said earlier in post that 6.6.13 work, Do you still have this kernel binary and its corresponding /lib/module/[whatever-if-any]6.6.13[whatever-if-any] directory and its modules? If you do I suugest you try to setup using the newly build zfsbootmenu and select the 6.6.13 kernel to see what happen. I hope this will help distinguish that error is in runtime kernel as opposite to zfsbootmenu kernel.

I have misunderstood your reference on docker. I assume you mean you were follow zfsbootmenu document that you were trying to build zfsbootmenu system in a docker container.

kernel's docker configuration is very unlikely at play during kernel initialisation. unless you did some special patching for docker for your runtime kernel it cannot be the cause.

display driver module "amdgpu" possible causing kernel panic, however I think this is not the case otherwise you should be able to get a glance of kernel panic on screen before it reboot. Question, have you try to boot without the "amdgpu"?

Suggestion,
Now that we agree on that there is two kernels in play, I suggest you consider make a very simple kernel for zfsbootmenu only and a real configuration for your everyday runtime kernel. for most of time the system update should only change the runtime kernel, You can leave the zfsbootmenu kernel for long time until you change zfs file system version (or something new in zfs in you are interested)
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Fri Apr 19, 2024 8:01 pm    Post subject: Reply with quote

kucklehead,

A new thoughts occur to me, your 6.6.21 kernel may not work because it is EFI stub kernel. I think kexec do not support PE32 header in the EFI stub kernel. (this is my current theory, still researching this)

If your 6.6.13 kernel image file still exist can you try this command to verify the image file format
Code:
file /path/to/<kernel 6.6.13 image file>


The "file" command will examine the header of the file for information to report type of file.

So please do the same for 6.6.21 kernel image file.

please share output of "file" test on both image files.

I hope this will help us understand better.

Thanks.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum