Question: how kexec?

[leyvi]

Hello Gentoo community.
I heard about this cool thing called `kexec` which allows you to (from my understanding) reload the kernel without rebooting. This is a really cool idea, but what I've discovered is that it's actually really inconvenient. The only practical way I've found of using `kexec` is through the `reboot` command, using the `-k` option. When I tried it, it didn't work too well, probably didn't configure it right, but that's not why I'm here. Is there a better way to do the same thing?
What I really want to do is this: with one command, hot-swap the kernel currently in RAM for a different one.
Not a problem that I need solving per-se, but I'd like some advice.

[gentoo_ram]

Kexec is not about hot-swapping kernels. All processes and hardware state are maintained by the current running kernel. If you start a new one, all that needs to be done again by the new kernel. So all processes need to go down and back up again. Maybe you save a couple seconds by not re-running EFI but you're not going to save much more than that. It's just like rebooting your machine either way.

I can only think of a couple scenarios where kexec might be useful:
1. Crash dumps. Maybe start a new minimal kernel that is just there to help perform a crash dump from memory to disk for kernel debugging.
2. Linux kernel in firmware of an embedded device wants to load a new kernel off storage.
3. Chain loading kernels. Maybe full-disk encryption? Boot a non-encrypted kernel and get enough authentication to be able to un-encrypt storage containing the "true" kernel.

There are other projects looking into hot-swapping but I don't think it involves Kexec.

[grknight]

kexec is much more useful in the server hardware space where reboot times can be literal minutes to probe everything.
I would like to have this work for me, but have not had success on my servers. (Use case being using a new kernel version, it always loads the old one)

[leyvi]

Alright, thanks for the insight everyone!
I understand it a lot better now.
Just to clarify: there is no way that I can hot-swap my kernel?
Maybe that should be a feature.

[grknight]

Alright, thanks for the insight everyone!
I understand it a lot better now.
Just to clarify: there is no way that I can hot-swap my kernel?
Maybe that should be a feature.

You cannot hot-swap a kernel because it is running the disks and other hardware and running processes depend on it. It would be like swapping aircraft mid-flight.

You can hot-patch a kernel using special tools. Patching a live kernel is not for the feint of heart and requires extremely tight conditions including the build tools being the same and the kernel minimally changed.

[leyvi]

Maybe the kernel devs should change that.
I'll bet there's a bunch of important structures in the kernel that represent hardware, processes, memory, etc and that's (from my understanding) the limiting factor. I wonder: Rust has these traits called `From<>` and `Into<>`, maybe we could make these critical structs somehow polymorphic (or more realistically, make it so that one struct can be made `::from()` the equivalent struct from a different kernel version). I imagine that these structs rarely change that much. Pointers to these structures could be passed to the new kernel, which could move them out of the way, somewhere else in kernel space, while it initializes, then these structs could be morphed into the proper types for that kernel version. Just an idea.
While all this happens, I seem to remember a signal (was it `SIGHALT`?) that can "pause" a process, maybe this could be used to block all processes in userspace while the kernel hot-swaps?
I seriously hope this isn't completely deluded.

[gentoo_ram]

Is it theoretically possible? Sure, I suppose so. But it would be a pretty complicated engineering effort. And given the number of platforms the Linux kernel runs on, very, very tricky. It would take hardware support which might only be possible on some specific hardware platforms. The question is: who would use this and would they rather have other solutions that might be more useful?

Obviously, high availability servers would be an interested demographic. But maybe they'd be more interested in engineering effort getting spent on more distributed computing type projects. Think clusters of computers working together. That might give you higher overall service availability.

Or maybe you'd treat it more like when you suspend a computer and bring it out of sleep. Except you suspend with one kernel version, write state to swap, block all hardware operations, then restart with a new kernel and resume the state from non-volatile storage. It wouldn't be hot-swapping, but it could be faster than a full reboot.

[leyvi]

Huh.
All these ideas are certainly interesting.
I hope a kernel dev stumbles across this thread, I think that if these features were included in the kernel, it could be nice quality of life improvement for everyone. Obviously, like gentoo_ram said, high-availability servers, mainframes, and supercomputers could be very interested in this, but also, think of it like this: more mainstream distributions like Fedora could integrate it with their package managers to allow for full system updates without a reboot. Think of it: just run `sudo dnf upgrade` (or whatever they do) and your kernel automagically gets upgraded, no reboot required. Could be great for small security updates or performance optimizations.
They would probably want to implement it on AMD64 first, then ARM64, then RISCV64, given those are some of the most used architectures. More obscure architectures like MIPS and x86 could come later.
Would also be nice to have on Android. Kernel version bumps on GrapheneOS always make my phone reboot way slower.

[Goverp]

... What I really want to do is this: with one command, hot-swap the kernel currently in RAM for a different one.
Not a problem that I need solving per-se, but I'd like some advice.

As others have said, it's not hot-swap, it's reboot without the BIOS/EFI stage. The new kernel does not have to be the same as the previous one, but it does have to be Linux.
On my desktop that's something between 5 and 10 seconds off a 20-40 second boot (I'm guessing, on another box, and no stop watch...). It's useful when running "git bisect", when the time saved accumulates!
To change kernels, using the Gentoo setup for OpenRc (I think it should also work for Systemd, but I don't use that), you configure /etc/kexec.conf, start the kexec service, and then reboot. I think there's some way to pass the name of the desired new kernel in the service start, but I've not tried it.
Alternatively, you should be able to load a new kernel with "kexec -l <kernelname> ...." with other appropriate parameters, and then issue "reboot -k".

[Hu]

Kpatch and similar mechanisms for applying minor (usually security) fixes to a running kernel were already years in the making have substantial limitations on when they can be used. A hot kernel upgrade would avoid the need to use those tools, so it seems reasonable to think that if hot kernel upgrade were easy, that would have been developed instead of these limited and finicky mechanisms. I think gentoo_ram's comment is on point that this is a massive amount of work. Even user-space checkpointing of individual programs is rarely done due to the workload and ongoing test requirements, and that would be a substantially easier project than swapping out a running kernel.

I expect the kernel developers have at least idly thought of this years ago, and promptly written it off as a massive amount of trouble for the gains it would bring them.

[leyvi]

Yeah, Hu is probably right.
Still fun to think about though.

[gentoo_ram]

Ironically, this type of kernel hot-swap would probably be more feasible using more of a micro-kernel type of design at least in theory. As long as the dependencies between the different parts of the kernel were minimized then I bet you'd be able to, say, replace some problematic hardware driver relatively independently of other parts of the kernel. Or maybe there's a memory bug in the IPv4 TCP packet parser. Maybe you could replace just that component. That's what a micro-kernel is all about.

I say ironically because Linus himself famously railed against the micro-kernel design as an acedemic computer scientist's fairy tale of inefficiencies. And thus we are left with the "optimized" monolithic kernel that Linux is today that requires a restart in order to update.

[hujuice]

kexec is much more useful in the server hardware space where reboot times can be literal minutes to probe everything.
I would like to have this work for me, but have not had success on my servers. (Use case being using a new kernel version, it always loads the old one)

I was using kexec years ago, with easy success. I don't have any memory or saved configurations now.
HUjuice

[Goverp]

I had it broken for some time, turning out to be a bug in kexec-tools on AMD Zen 3 cpus that was recently fixed. I had it working with the patch from that bug, but not as of the last time I updated my system (a couple of weeks ago) without the patch.