[SOLVED] mdadm array MIA after kernel rebuild

[statikregimen]

UPDATE: Yeah, I'm stuck on this one. Have replied with more information....When I made this post, I couldn't find a USB drive to boot the machine, so took a chance in posting while I sorted that, thinking maybe there was a change in genkernel, grub, or something else that I was overlooking. That's why extra details are in my first reply...

Oof...Hello all,

After a seemingly harmless kernel rebuild, my array isn't coming up, causing a kernel panic since it holds my root partition. To rebuild the kernel, I'm using the same commands I always have. Nothing has changed in the config - I only rebuilt because I accidentally deleted the working kernel from my boot partition while trying to clean up some older ones. Was wondering if anything related changed with genkernel (used to build initramfs) or grub that I'm overlooking.

Also, I first noticed this issue a while back when trying to update from 5.4.48 to 5.4.60, so I was using 5.4.48 for a while until I had a chance to sort 5.4.60...

Thank you, as always, for reading! Cheers.

[Whissi]

When you are using genkernel to only create initramfs, are you passing --kernel-config to genkernel? Please show us your kernel command-line, too.

[statikregimen]

Thanks for the reply! Was able to borrow a USB drive, so here is some more infos...

Here is a pic showing the kernel panic:
https://meaninglessinfo.com/IMG_20200922_220204977.jpg

These are the commands I use to rebuild the kernel each time:

Code: Select all

kergen -u
mount /boot
cd /usr/src/linux
make && make modules_install && make install
genkernel initramfs --mdadm
grub-mkconfig -o /boot/grub/grub.cfg

From the screenshot, looking at the lines...:

Code: Select all

...
md: Skipping autodetection of RAID arrays. (raid=autodetect will force)
VFS: Cannot open root device "UUID=f8ea...." or unknown-block(0,0): error -6
Please append a correct "root=" bootoption; here are the available partitions:
...

...I can only speculate that grub-mkconfig is going wrong or genkernel is not building the initramfs properly? Idk how else to explain this, when using such automated tools.

Looks like all the files I'm used to seeing are being generated and are present on my boot partition. Using an Ubuntu Live USB, I've also been able to verify that the array is healthy and partitions/data in-tact.

[Whissi]

I see two problems:

1. When you don't build both, kernel and initramfs, with genkernel, you must tell genkernel which kernel config you used or generated initramfs won't match used kernel. To do that, pass --kernel-config=/path/to/kernel.config to genkernel when only creating initramfs.
2. The screenshot doesn't show anything from genkernel. You are only seeing errors from kernel. Looks like you didn't boot genkernel initramfs.

[statikregimen]

So this is a documented recent change in Genkernel? I can't stress enough how routine this is (or rather should have been) for me, and always have used the same exact commands. It seems silly for Genkernel to suddenly not use /usr/src/linux/.config by default, when no config is specified. I will nevertheless give it a try when I get home.

Thanks!

[Whissi]

No, it isn't a changed behavior. Just pay attention to genkernel output. It will always tell you which kernel.config it is using.
But it's complicated:

Genkernel first tries to determine KV from specified kernel sources, /usr/src/linux by default.
Based on read KV, it will look in /etc/kernels/ for a kernel config matching that version (because genkernel by default will save used kernel.config to that directory after a successfully build).
If it cannot find a config in /etc/kernels matching determined KV it will fall back to generic kernel config provided by genkernel. That's for example the case when you build a new kernel for the first time on that system without manually specifying a kernel config.

Only if you pass --no-mrproper you will trigger a special behavior which will make genkernel look for any existing .config in specified KOUTPUT_DIR (by default this is also /usr/src/linux but keep in mind you can also build kernel into a dedicated directory to leave sources untouched...).

I really don't understand why people expect that genkernel would pick up anything from an arbitrary CWD without telling...

Also keep in mind that it only makes limited sense to build kernel on your own but still use genkernel to create an initramfs. For example, genkernel tries to ensure that features you asked for are enabled in kernel. In recent genkernel versions we fixed compression support (--compress-initramfs-type=best|fastest didn't worked). After the fix went stable, people not paying attention to used kernel.config when generating initramfs ended up with an initramfs not supported by the kernel they booted because genkernel fall back to generic kernel config which supports a better compression algorithm which were picked during initramfs generation.

[statikregimen]

Thank you for the clarification. I didn't have any expectations about genkernel using CWD, but rather expected it would use /usr/src/linux/.config by default (well, before this I didn't even realize it needed the config for this). Seems like a perfectly reasonable assumption to me. Basically the reason I'm doing it like this, is I like to have a lean kernel and enjoy the hands-on experience of the config process. However, it wasn't until I had already done all the work that I realized I'd also need an initramfs if I wanted to set my drives up the way I wanted. At that point, was just anxious to get the install finished so integrated genkernel into the process to save time. Been using the same kernel config & bash script ever since.

Again, thank you very much for the help. I should have it working as soon as I get home, and will mark solved when I'm sure I don't have any more moronic oversights

Solution:
I can still only loosely speculate as to what went wrong that suddenly required this new thing, but the solution for me is to specify --kernel-config to genkernel, thusly:

Code: Select all

genkernel initramfs --mdadm --kernel-config=/usr/src/linux/.config

It also fixed my issue with 5.4.60