Attempting to fix genkernel with merged-usr.

[tlhonmey]

Genkernel still works perfectly with split-usr profiles. But when trying to use it from a merged-usr profile, anything which is not statically-compiled fails to run with a rather unhelpful "sh: not found" error.

In my case this specifically messes up Plymouth... See Bug But, based on the results of attempting to figure that out, it's just any, non-static binary.

Unpacking the initramfs into a chroot and poking around gets exactly the same errors.

I did note that there's kind of a conflict between what genkernel sets up as the initial filesystem without /bin and /usr/bin (&c) being symlinked and what plymouth specifically creates in the overlay that genkernel tries to tack on later. So I figure something's being overwritten somewhere...

So I attempted to update genkernel to handle merged-usr layouts.

Code: Select all

--- genkernel-4.3.17/gen_initramfs.sh   2025-06-10 09:14:40.000000000 -0700
+++ /tmp/gen_initramfs.sh       2025-12-04 11:28:34.642865787 -0800
@@ -282,10 +282,14 @@

        mkdir "${TDIR}" || gen_die "Failed to create '${TDIR}'!"
        cd "${TDIR}" || gen_die "Failed to chdir to '${TDIR}'!"
-
+       print_info 5 "$(get_indent 2) Beginning construction of base_layout"
        local mydir=
        for mydir in \
                .initrd \
+               usr \
+               usr/bin \
+               usr/lib \
+               usr/sbin \
                bin \
                dev \
                etc \
@@ -298,15 +302,25 @@
                sbin \
                sys \
                tmp \
-               usr \
-               usr/bin \
-               usr/lib \
-               usr/sbin \
                var/empty \
                var/log \
                var/run/lock \
        ; do
-               mkdir -p "${TDIR}"/${mydir} || gen_die "Failed to create '${TDIR}/${mydir}'!"
+               if [ -L "/${mydir}" ]
+               then
+                       print_info 5 "$(get_indent 2) Detected symlinked folder ${mydir},"
+                       print_info 5 "$(get_indent 2) pointing at $(readlink /${mydir}),"
+                       print_info 5 "$(get_indent 2) cloning to ${TDIR}/${mydir}."
+                       cp -a "/${mydir}" "${TDIR}/${mydir}" || gen_die "Failed to clone link '${TDIR}/${mydir}'!"
+               else
+                       if [ ! -e "${TDIR}/${mydir}" ]
+                       then
+                               print_info 5 "$(get_indent 2) Creating folder ${TDIR}/${mydir}"
+                               mkdir -p "${TDIR}"/${mydir} || gen_die "Failed to create '${TDIR}/${mydir}'!"
+                       else
+                               print_info 5 "$(get_indent 2) ${TDIR}/${mydir} Already exists $([ -L "${TDIR}/${mydir}" ] && echo as symlink)"
+                       fi
+               fi
        done

        chmod 1777 "${TDIR}"/tmp || gen_die "Failed to chmod of '${TDIR}/tmp' to 1777!"
@@ -1364,6 +1380,8 @@
        # can probably get rid of this; depends if plymouth was built with static libs
        # rm -f "${TDIR}"/lib*/{ld*,libc*,libz*} \
                # || gen_die "Failed to clean up plymouth cpio archive!"
+       #just the opposite really, I think.  Plymouth is hard to build static.  Take no chances.
+       touch "${TEMP}/.binaries_copied"

        ln -sf "${PLYMOUTH_THEME}/${PLYMOUTH_THEME}.plymouth" "${TDIR}/usr/share/plymouth/themes/default.plymouth" \
                || gen_die "Failed to set the default plymouth theme!"

These changes make the symlink structure in the initramfs match with what's on a merged-usr system's filesystem, and make sure that supporting libraries get copied.

With these changes, I can unpack the generated initramfs into a chroot, and all the dynamically linked executables seem to actually run.
But... now when I try to boot with the initramfs, I get nothing but a black screen for thirty seconds or so, and then the system reboots.
If I turn off "quiet" in the linux cmdline options, I get one message from the kernel about it finding the initrd in memory, and then nothing until it reboots itself.

Debugging this is, obviously, rather a challenge. Especially where it seems like it should work based on stuff working in a chroot... And especially where it was working previously with nothing but the stuff genkernel is statically compiling, which definitely seems to all still be there. So I'm kind of working completely blind.

So I'm looking for a bit of help from somebody who has a bit more experience with initramfs stuff. I can probably even pay a reasonable fee if there's somebody who wants to just "make it work" and deal with getting it into an official release.


And because I know a bunch of people are going to jump in and just tell me to use something else, to do so I require the following:
1) Plymouth support or similar, animated splash screen that loads early and works regardless of display resolution.
2) The ability to use the same initramfs to boot either from read-only recovery media like a DVD, or from the installed system.
3) An easy way to patch in an equivalent to my custom boot method that loads the contents of a tarball into a zram for recovery purposes rather than relying on squashfs. (No, I'm not including this in my testing builds, and it doesn't get that far anyway.)

Dracut fails #2 last time I tried it. Probably fails #3.
ugrd fails #1. Probably fails #2.
If you've got a suggestion that might be less work than just fixing genkernel, I'd be happy to look at it. But please don't clutter things up with pointless warnings about genkernel not being well-maintained. I already know that.

[zen_desu]

And because I know a bunch of people are going to jump in and just tell me to use something else, to do so I require the following:
1) Plymouth support or similar, animated splash screen that loads early and works regardless of display resolution.
2) The ability to use the same initramfs to boot either from read-only recovery media like a DVD, or from the installed system.
3) An easy way to patch in an equivalent to my custom boot method that loads the contents of a tarball into a zram for recovery purposes rather than relying on squashfs. (No, I'm not including this in my testing builds, and it doesn't get that far anyway.)

Dracut fails #2 last time I tried it. Probably fails #3.
ugrd fails #1. Probably fails #2.
If you've got a suggestion that might be less work than just fixing genkernel, I'd be happy to look at it. But please don't clutter things up with pointless warnings about genkernel not being well-maintained. I already know that.

If you're remotely interested in debugging some of the ugrd stuff, I've been working on a branch to try to improve plymouth support: https://github.com/desultory/ugrd/compa ... ..plymouth

I just rebased this branch, and you should be able to try to force debugging with `plymouth_debug = true` in your ugrd.config. If you want to try to make this work better, this option is mostly setting this: https://github.com/desultory/ugrd/compa ... 35cebdeR63

It's still not working (doesn't work with themes) and I think it's because plymouth will not function properly without udev. I can't get plymouth to properly share debug info no matter how I build it, and have tried forcing the GPU module to be activated early and things like simpledrm instead of efifb, but simply cannot get plymouth to realize there is a GPU available. I think if plymouth can get this "signal" (however it exists) things will just work, but I haven't been able to coerce it to work in a VM or on my laptop.

for point 2, ugrd has a module for livecds, but currently it's implemented as an entirely alternate route for the boot mounts:

https://github.com/desultory/ugrd/blob/ ... /livecd.py

This could be changed so it will let you add that module and only do that stuff if you provide init args like "ugrd_livecd_label" for the storage the livecd squashfs lives on and "ugrd_squashfs_image" is used to specify the path of the squashfs you want to use for the live environment.

As it is, this module is mostly a proof of concept so ugrd could be used by releng for official ISOs if they want, so the defaults are opinionated towards that use case.

[Hu]

Regarding debugging initramfs, as zen_desu hints at briefly, virtual machines are a huge improvement here. You don't need to take down your main machine just to test a candidate build. You can swap in new candidates far more readily. You can easily get serial console output from a guest, which may offer a viable way to extract debugging information. The only notable downside is that the guest will be using a different graphics stack, so success or failure running a bootsplash in a guest might not predict whether it will also work on real hardware. Even so, if I were working on this, I would iterate in a qemu guest until I had something working, and only then switch over to trying it on real hardware.

[zen_desu]

Regarding debugging initramfs, as zen_desu hints at briefly, virtual machines are a huge improvement here. You don't need to take down your main machine just to test a candidate build. You can swap in new candidates far more readily. You can easily get serial console output from a guest, which may offer a viable way to extract debugging information. The only notable downside is that the guest will be using a different graphics stack, so success or failure running a bootsplash in a guest might not predict whether it will also work on real hardware. Even so, if I were working on this, I would iterate in a qemu guest until I had something working, and only then switch over to trying it on real hardware.

Yeah, 90% of the debugging I've done has been in a VM but plymouth can be very "weird" in how it behaves with different graphics hardware, so it's important it's tested on different hardware. I use a laptop to do this because it boots faster than my desktop and I do less work on it. More than half of the time spent debugging this stuff is just rebooting which can be really painful. I've considered trying to add support for multiple TTYs just to try to possibly make debugging it at runtime more reasonable. The main issue I've had with plymouth is that the only way I've been able to get any info from it has been to try to strace it in the initramfs, which is a mess because plymouth can break if anything is written to the TTY it wants to use

[NeddySeagoon]

tlhonmey,

genkernel is depreciated and will be going away 'soon'.
It's already been removed from the handbook.
Before you put more effort into genkernel, ask yourself if you are prepared to maintain genkernel going forward?

Long term your time may be better spent contributing to fixing one of the other options, even though its more work right now.

Is hand rolling your kernel and initrd an option?
You build the kernel so that all the kernel modules required to boot are built into the the kernel. That's important.
The initrd is now reduced to a collection of user space utilities.
Now that there are no kernel modules in the initrd, there is no need to keep rebuilding it every kernel update.
It becomes like firmware. Build it once for the life of the hardware.
This won't work if the kernel and initrd must boot anywhere.

[tlhonmey]

If you're remotely interested in debugging some of the ugrd stuff, I've been working on a branch to try to improve plymouth support: https://github.com/desultory/ugrd/compa ... ..plymouth

Yeah, ugrd definitely looks like it's the alternative that's the most tractable for my esoteric usecase. It's just still going to be a lot of work, so I was hoping to figure this out so I have some breathing-room. Especially where it seems like it should be relatively simple...

With regard to Plymouth, I can tell you that yes, it definitely requires udev. Like, it pokes at some udev internal stuff for some reason that I don't understand and won't even try to run if it doesn't find it. I might have patches to remove that which may or may not still work lurking around here somewhere... But putting udev into the initramfs is probably the more maintainable option...

And, also, you have to both compile it with the debug config option, and then also launch every piece of it with the --debug option in order to get any debug output out of it. It's really unfriendly in that regard.

Also, the easiest way I've found to debug Plymouth in the initrd is to compile your busybox with the telnetd applet enabled. That gives you an easy way to connect in without disturbing Plymouth's hold on the display. Dropbear or a serial terminal would also work, obviously, but can be more fiddly to get working right.

Before you put more effort into genkernel, ask yourself if you are prepared to maintain genkernel going forward?

Long term your time may be better spent contributing to fixing one of the other options, even though its more work right now.

Yeah. There is that. I'm less worried about the amount of work involved in switching than about needing something functional while I do it. And if I can get genkernel's initramfs builder happy with the new profile, then it should work for a good, long time while I explore alternatives.

Is hand rolling your kernel and initrd an option?
You build the kernel so that all the kernel modules required to boot are built into the the kernel. That's important.
The initrd is now reduced to a collection of user space utilities.
Now that there are no kernel modules in the initrd, there is no need to keep rebuilding it every kernel update.
It becomes like firmware. Build it once for the life of the hardware.
This won't work if the kernel and initrd must boot anywhere.

It doesn't need to boot anywhere. But it does need to boot on more than one piece of hardware... Just rolling my own is something I have considered, but there are a few of the more complex things like mdadm, etc. that I do actually need to look for and enable (or not) correctly. So using scripts that have more eyes on them than just mine is beneficial for reliability purposes...

But your suggestion did give me an idea for a fallback option... Because the only thing really changing between iterations of the initramfs is the kernel modules... So, if nothing else, I can probably just take my last, working, split-usr initramfs, rip out the modules, and put in the new ones as-needed... I'm sure there will be some issues with that too, but it'll probably get me by until I can sort out something to do with ugrd.


Since it's not even getting to running plymouth, I'm going to try setting up a VM with serial console. It feels like it's really close to working, and there's just some, trivial glitch that I'm overlooking. And if I can get that ironed out then that should buy me at least a year or two to make something else work.

[NeddySeagoon]

tlhonmey,

You can have several initrd files so how about a hybrid idea?
Have a userspace initrd and a seperate kernel-modules initrd file?
You may need to update the init script from time to time.

Some tools already create a separate microcode initrd.

[tlhonmey]

That's not a bad idea, although since the cpio format for the initramfs is appendable I'm not sure there's much effective difference between that and just keeping a base initramfs with all the fancy graphics stuff and boot scripts and then just appending the latest modules whenever I whack one of these together.

[pingtoo]

The multiple cpio archives is one idea I been contemplating for some time.

compare to update one main cpio archive with appendage is that multiple cpio is a deferred operation. It no longer require one to remember to execute a operation at the time of post building. it defer to either at time of boot configuration or at bootloader or inside initramfs play.

for example, grub boot loader script can be program to automatically load if a second cpio archive exist and load it when boot. It is also possible to use the /init script in initramfs to find the secondary cpio archive and load it before module probe.

It become fancier if you do network boot then many of these operations can be network loaded which give grate convenient and possible easy recovery when something gone unexpectedly.

My hesitation was because u-boot does not work will for this idea (at least last time I check it does not support load more than one initrd). And currently the UEFI spec does not support loading second initrd. at lease the UEFI shell script is bit difficult for me at this time.