Load order of kernel modules and firmware with initramfs

[nagmat84]

How, when and in which order does the Linux kernel load modules (and the respective firmware) especially if an initramfs is used? I have searched for a definitive guide which considers all options systematically, but haven't found one. Maybe the answer should included in a subsection below Gentoo Handbool: Kernel - Alternative: Manual configuration.

As far as I understand, there are 5 options:

1. No initramfs, drivers statically linked into kernel, required firmware embedded in kernel
2. No initramfs, drivers as loadable modules, required firmware on real root
3. Initramfs, drivers as loadable modules in initramfs, required firmware in initramfs
4. Initramfs, drivers as loadable modules in initramfs, required irmware on real root
5. Initramfs, drivers as loadable modules on real root, required firmware on real root

My thoughts:

• Option 1: OK. Firmware must mandatory be embedded into kernel, because statically linked drivers may be loaded before the root fs becomes available. Is that statement correct?
• Option 2: OK. The kernel loads modules after the root fs has become available. So it is fine to have the firmware on real root.

I read the statement "the kernel loads modules after the root fs has become available" on several sites. However, it was never clearly stated whether root fs refers to the temporary root fs provided by the initramfs or if this refers to the final, real root fs.

Here are my thoughts on the remaining three options:

• Option 3: Does the kernel load modules from the initramfs or does it needlessly increase the size of the initramfs? Until now I thought the former is correct.
• Option 4: I assume this option is simply bogus. Either the kernel loads modules from initramfs before the real root is available, but then the firmware is missing. Or the kernel does not load modules from initramfs, but then there is no point in putting the modules into the initramfs in the first place. (Note, of course this option makes sense for drivers without firmware, but then option 3 and 4 are identical.)
• Option 5: Will the kernel still load additional modules from the real root after the kernel switched the root fs or is it too late, because the kernel has already probed all hardware? Note, of course the kernel will load additional modules for hardware that will be attached to the computer at a later point (e.g. a USB stick). I am referring to drivers for hardware which is already there when the computer powers up.

Is even a mix of option 3 and 5 possible? I.e. have some modules and their firmware in the initramfs and some modules and their firmware on the real root? (Again I am talking about built-in hardware, not removable hardware which is plugged-in later.) Assume I had a LUKS-encrypted root fs and I use an initramfs to get a nice boot splash and password prompt with Plymouth. Would the following setup work? Put the graphics driver module and its firmware into the initramfs, but keep the modules for wifi, bluetooth, camera etc. and there firmware on the real, encrypted root fs. Will they still be loaded after the real root fs has been unlocked?

[zen_desu]

1. Yes, if you build a kmod into the kernel, you must include firmware, as it needs that as soon as it loads that module.
2. Yes, in this case modules required to mount the root must be builtin, but modules which need firmware will load it from the root as you'd expect.
3. Any modules can be included in the initramfs, and depending on the initramfs design, some or all may be loaded. Anything which needs firmware must have the firmware in the initramfs.
4. Technically possible, but requires the initramfs to mount the root, then read files from it and use those, it's not easy and makes almost no sense to attempt.
5. The kernel will use the initramfs environment for loading those modules/firmware while in the initramfs, and from the "real" root after switch_root is performed. You could include kmods in the initramfs lacking firmware, and they may load to some degree, but there will likely be issues. In most realistic cases, firmware is only needed after the root is mounted, unless you're doing something strange like NFS root over WiFi where the NIC requires firmware.

Plymouth unfortunately seems to require full GPU drivers to function. You can get it to "run" without them, but you get a plain grey screen. If you're willing to make an initramfs that is >100mb uncompressed just for a splash screen, that works, but I consider this to be a waste of space. You also have to consider that the system must read and decompress that entire image, this will almost certainly slow down your boot process at least at bit once you start to have very large images, especially if stored on a USB.

One thing to consider when copying kmods is that you must re-run depmod once done, unless you copy all kmods. If you copy all kmods, be sure to copy the whole kmod dir with the metadata files.

[pietinger]

[...] As far as I understand, there are 5 options:

1. No initramfs, drivers statically linked into kernel, required firmware embedded in kernel
2. No initramfs, drivers as loadable modules, required firmware on real root
3. Initramfs, drivers as loadable modules in initramfs, required firmware in initramfs
4. Initramfs, drivers as loadable modules in initramfs, required irmware on real root
5. Initramfs, drivers as loadable modules on real root, required firmware on real root

My thoughts:

• Option 1: OK. Firmware must mandatory be embedded into kernel, because statically linked drivers may be loaded before the root fs becomes available. Is that statement correct?
• Option 2: OK. The kernel loads modules after the root fs has become available. So it is fine to have the firmware on real root.

Yes ... and ... Yes.

If kernel has an initramfs:

It's easy if you know what the kernel does when an initramfs is present: The kernel initializes all modules that are statically included in the kernel AND then passes control completely to the init-script. It does NOT try to mount the root partition. This is NOW the task of the init-script. There is no way back from the init-script to the kernel. The last thing the init-script does is to start /sbin/init from the real root partition.

Now it depends on WHAT the init-script does in WHAT ORDER. Normally it will try to mount the root partition as soon as possible. If it needs kernel modules for this - which are not yet statically initialized - these modules must be present IN the initramfs. SHOULD one of these modules require firmware, this MUST also be present in the initramfs (this is your option 3). For all kernel modules that are loaded AFTER mounting the root partition (=usually everything else - except the modules that are necessary to access the root partition) the same applies as for a system without initramfs: They can be loaded normally and do not have to be present in the initramfs (this is your option 5).

Option 4 is NOT possible ... please keep in mind, that not every kernel module needs firmware.

Is even a mix of option 3 and 5 possible?

YES.

You must put everything INTO the initramfs what is necessary to access the real root fs ... IF it is NOT statically configured in the kernel.

Everything else can be on the real root ... IF the init-srcipt initializes these modules AFTER it has mounted the root.

[nagmat84]

So, just to make sure that I understood everything, I'll reprase in my own words:

When the kernel discovers a device, the kernel doesn't load the corresponding module itself, but reports the event to user space. A user-space process (typically the udev deamon) has to pick up that event and load the appropriate module.

Now, let's assume that udev is included in the initramfs (because that is what dracut does as far as I know). Let's also assume that the initramfs includes some drivers (but not all) as modules. Is the following true?

1. The kernel probes the buses and finds the devices.
2. The kernel initializes the drivers which are linked statically.
3. The kernel mounts the initramfs as the (temporary) root fs
4. The kernel starts the init process from the initramfs
5. The init process from the initramfs starts udev
6. The udev daemon (the one from the initramfs) loads all kernel modules for devices which the kernel reported and are included in the initramfs (obviously it cannot load modules which are not included)
7. The init process of the initramfs changes to the true root fs and starts the real init process
8. The (real) init process starts udev (this time from the final root fs)
9. The udev daemon (re-)reads the list of kernel events for discovered devices and loads the modules which haven't been loaded yet

[zen_desu]

I'm not sure how udev works top to bottom, but it more or less waits for kernel device events, and based on the loaded rules ( /usr/lib/udev/rules.d, /etc/udev/rules.d/ ), it executes actions in response to those events.

If you want a much simpler example, ugrd just does this (on my machine):

Code: Select all

modprobe -av dm_crypt uhid uas crc32c vfat nvme

udev is not used, so it just runs this command early in the init process.

required kmods for that system/kernel are determined at build time.

[pietinger]

[...] Is the following true?

1. The kernel probes the buses and finds the devices.
2. The kernel initializes the drivers which are linked statically.
3. The kernel mounts the initramfs as the (temporary) root fs
4. The kernel starts the init process from the initramfs
5. The init process from the initramfs starts udev
6. The udev daemon (the one from the initramfs) loads all kernel modules for devices which the kernel reported and are included in the initramfs (obviously it cannot load modules which are not included)
7. The init process of the initramfs changes to the true root fs and starts the real init process
8. The (real) init process starts udev (this time from the final root fs)
9. The udev daemon (re-)reads the list of kernel events for discovered devices and loads the modules which haven't been loaded yet

I am very unsure if in an initramfs udev is in use (5 + 6). TBH I am too lazy / have no time to install the current gentoo-kernel-(bin) and inspect its initramfs ... AFAIK it is built with "dracut" ... and you can inspect it easily with: https://wiki.gentoo.org/wiki/Custom_Ini ... io_archive ... I guess it is working with "modprobe" also ... but I don't know for sure.

[zen_desu]

[...] Is the following true?

1. The kernel probes the buses and finds the devices.
2. The kernel initializes the drivers which are linked statically.
3. The kernel mounts the initramfs as the (temporary) root fs
4. The kernel starts the init process from the initramfs
5. The init process from the initramfs starts udev
6. The udev daemon (the one from the initramfs) loads all kernel modules for devices which the kernel reported and are included in the initramfs (obviously it cannot load modules which are not included)
7. The init process of the initramfs changes to the true root fs and starts the real init process
8. The (real) init process starts udev (this time from the final root fs)
9. The udev daemon (re-)reads the list of kernel events for discovered devices and loads the modules which haven't been loaded yet

I am very unsure if in an initramfs udev is in use (5 + 6). TBH I am too lazy / have no time to install the current gentoo-kernel-(bin) and inspect its initramfs ... AFAIK it is built with "dracut" ... and you can inspect it easily with: https://wiki.gentoo.org/wiki/Custom_Ini ... io_archive ... I guess it is working with "modprobe" also ... but I don't know for sure.

I'm 99% sure dracut always tries to use udev, I don't think it's used for the majority of module loading.

I tried to look into whether or not udev loads kmods and could not find any example of it doing on my system, or having a mechanism to. I could be missing something, I've never been able to entirely figure out udev. I don't think it's a necessity within an initramfs especially if you know what is expected on booting system.

[GDH-gentoo]

1. The kernel probes the buses and finds the devices.
2. The kernel initializes the drivers which are linked statically.
3. The kernel mounts the initramfs as the (temporary) root fs
4. The kernel starts the init process from the initramfs
5. The init process from the initramfs starts udev
6. The udev daemon (the one from the initramfs) loads all kernel modules for devices which the kernel reported and are included in the initramfs (obviously it cannot load modules which are not included)
7. The init process of the initramfs changes to the true root fs and starts the real init process
8. The (real) init process starts udev (this time from the final root fs)
9. The udev daemon (re-)reads the list of kernel events for discovered devices and loads the modules which haven't been loaded yet

I believe that this is mostly correct for an initramfs that includes udev, which I'm also quite sure is what dracut produces. With a small clarification about 9)

I'm not sure how udev works top to bottom, but it more or less waits for kernel device events, and based on the loaded rules ( /usr/lib/udev/rules.d, /etc/udev/rules.d/ ), it executes actions in response to those events.

I think that this is basically what happens, yeah.

There is a clarification to make: module loading by udev happens only while the udev daemon is running and listening for events, obviously. One can make the kernel (re-)emit all relevant events with udevadm trigger (mdev -s does the same for mdev); that's what's done in step 9 by the init system. For OpenRC that's the udev-trigger service's job (from package sys-fs/udev-init-scripts).