[solved] dump-capture kernels and initramfs (kexec -p)

Message

sublogic · Post by **sublogic** » Thu Mar 24, 2022 7:19 am

I use lvm over LUKS to encrypt root, /var, /home and swap. The initramfs came from genkernel.

I'm trying to capture a crash dump after a kernel panic (see this topic). I read the kdump wiki article. I had a lot of trouble with video modes but I finally got "kexec -l" to work with the following:

Reboot.
Press "e" in grub to edit the chosen entry.
Comment out the set gfxpayload=keep that came from my /etc/default/grub.
In the "linux" line, add options "radeon.modeset=0 init_opts=S"

Taking out "gfxpayload" makes sure the console starts in ugly 80x25 VGA mode.
The "radeon.modeset" thing sabotages my graphic chip's kernel module and keeps the console in ugly VGA. This is of course specific to my machine.
The "init_opts" is genkernel initramfs syntax to stay in single-user mode so we don't try to start X (which would fail because radeon didn't load).
Press F10 to boot the entry.
Enter the root password for single-user mode.
Mount /boot. Optionally remount / read-write.

Run "./kexec.sh -l", this script.

Code: Select all

#! /bin/bash

usage() {
    echo Usage: "$0 -l|-p"
}

case $1 in
    -l|-p) kopt=$1;;
    *) usage; exit 1;;
esac

cmdline="root=/dev/vg0/root debug ro dolvm crypt_root=/dev/sda3"
cmdline="$cmdline hpet=force resume=/dev/vg0/swap init_opts=S"
cmdline="$cmdline irqpoll maxcpus=1 reset-devices"

set -x
kexec $kopt /boot/vmlinuz-5.15.26-gentoo-x86 \
    --initrd=/boot/initramfs-5.15 \
    --reset-vga --console-vga \
    --append="$cmdline"

sync; reboot.

I use a script to be sure that I call kexec consistently. The script is also specific to my machine. The only parameter to the script is whether to run "kexec -l" (warm reboot) or "kexec -p" (capture kernel). I don't know if the "--reset-vga --console-vga" is necessary but it didn't hurt.

Well, the warm reboot works perfectly. No POST, go grub, the kernel restarts immediately, the initramfs loads, I get a rescue shell ("debug" in $cmdline), I exit the rescue and type my LUKS passphrase, I exit the next two rescue shells and I end up in single-user mode with the ugly VGA. Success! Yea. Now for the crash test. The capture kernel will be the same as the crashing kernel.

Reboot.
Press "e" in grub.
Comment out the set gfxpayload=keep .
Add options "radeon.modeset=0 init_opts=S crashkernel=64M nokaslr"
Press F10. Play in the rescue shell, enter the LUKS passphrase, etc.
Mount /boot. Optionally remount / read-write.
Run "./kexec.sh -p" .
At this point I'm in single-user mode with a capture kernel preloaded.
sysrq-c to trigger a panic.

FAIL ! Panic (expected), capture kernel starts (good), but it panics immediately:

Code: Select all

Kernel panic - not syscing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU:0 PID:1 Comm: swapper/0 Not tainted 5.15.26-gentoo-x86 #1
Hardware name: Gateway Mx       /, BIOS 83.08 03/06/07
Call Trace:
 dump_stack_lvl+0x34/0x44
 dump_stack+0xd/0x10
 panic+0x95/0x24d
 mount_block_root+0xef/0x165
 mount_root+0x123/0x12b
 prepare_namespace+0x116/0x141
 kernel_init_freeable+0x1e8/0x1f5
 ? rest_init+0xa0/0xa0
 kernel_init+0x17/0x100
 ret_from_fork+0x1c/0x28
Kernel Offset: disabled
---[ end Kernel panic - not syscing: VFS: Unable to mount root fs on unknown-blcok(0,0) ]---

There, I typed it (minus the timestamps). Typos possible, but I was careful. Yes, it's a 15-year old laptop.

It's not telling me what root it was trying to mount. If it tried /dev/vg0/root that's not going to work. Lines above the panic and still on-screen made it clear that it saw /dev/sda3, my crypt_root. It's as if it skipped the early userspace. Did it ignore the initramfs ?

Any ideas ? Do capture kernels skip early userspace ? Do I have to build in a /dev/ram0 with a final rootfs image ? Or did I leave out something that made it miss the initramfs ?

Kernel config here. Expires 30 days from this post.
emerge --info below.

Code: Select all

$ emerge --info sys-apps/kexec-tools
Portage 3.0.30 (python 3.9.10-final-0, default/linux/x86/17.0/desktop, gcc-11.2.1, glibc-2.33-r7, 5.15.26-gentoo-x86 i686)
=================================================================
                         System Settings
=================================================================
System uname: Linux-5.15.26-gentoo-x86-i686-Intel-R-_Core-TM-_Duo_CPU_T2250_@_1.73GHz-with-glibc2.33
KiB Mem:      952964 total,    239408 free
KiB Swap:    2097148 total,   2097148 free
Timestamp of repository gentoo: Sat, 19 Mar 2022 13:00:01 +0000
Head commit of repository gentoo: 8c18029f63d9e89e12906f4e59e0c3b63cbf06ec
sh bash 5.1_p16
ld GNU ld (Gentoo 2.37_p1 p2) 2.37
distcc 3.4 i686-pc-linux-gnu [enabled]
app-misc/pax-utils:        1.3.3::gentoo
app-shells/bash:           5.1_p16::gentoo
dev-lang/perl:             5.34.0-r6::gentoo
dev-lang/python:           3.9.10_p1::gentoo, 3.10.2_p1::gentoo
dev-lang/rust:             1.58.1::gentoo
dev-util/cmake:            3.22.2::gentoo
dev-util/meson:            0.60.3::gentoo
sys-apps/baselayout:       2.7-r3::gentoo
sys-apps/openrc:           0.44.10::gentoo
sys-apps/sandbox:          2.25::gentoo
sys-devel/autoconf:        2.13-r1::gentoo, 2.71-r1::gentoo
sys-devel/automake:        1.16.4::gentoo
sys-devel/binutils:        2.37_p1-r2::gentoo
sys-devel/binutils-config: 5.4::gentoo
sys-devel/clang:           13.0.0::gentoo
sys-devel/gcc:             10.3.1_p20211126::gentoo, 11.2.1_p20220115::gentoo
sys-devel/gcc-config:      2.5-r1::gentoo
sys-devel/libtool:         2.4.6-r6::gentoo
sys-devel/lld:             13.0.0::gentoo
sys-devel/llvm:            13.0.0::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  5.15-r3::gentoo (virtual/os-headers)
sys-libs/glibc:            2.33-r7::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://192.168.1.50/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 2

ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="@FREE"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=pentium-m -mavx256-split-unaligned-load -mavx256-split-unaligned-store -mno-accumulate-outgoing-args -mno-sahf -msse3 -mtune=generic --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=2048"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=pentium-m -mavx256-split-unaligned-load -mavx256-split-unaligned-store -mno-accumulate-outgoing-args -mno-sahf -msse3 -mtune=generic --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=2048"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -march=pentium-m -mavx256-split-unaligned-load -mavx256-split-unaligned-store -mno-accumulate-outgoing-args -mno-sahf -msse3 -mtune=generic --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=2048"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distcc distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=pentium-m -mavx256-split-unaligned-load -mavx256-split-unaligned-store -mno-accumulate-outgoing-args -mno-sahf -msse3 -mtune=generic --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=2048"
GENTOO_MIRRORS="http://mirror.leaseweb.com/gentoo/ rsync://ftp.belnet.be/gentoo/gentoo/ http://gentoo.mirrors.tds.net/gentoo http://tux.rainside.sk/gentoo/"
LANG="C.UTF8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j7 -l3"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac acl acpi alsa bluetooth branding bzip2 cairo cdda cdr cli crypt cups dbus dri dts dvd dvdr elogind encode exif flac fortran gdbm gif gpm gtk gui iconv icu ipv6 jpeg lcms libglvnd libnotify libtirpc mad mng mp3 mp4 mpeg ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt5 readline sdl seccomp spell split-usr ssl startup-notification svg tiff truetype udev udisks unicode upower usb vorbis wxwidgets x264 x86 xattr xcb xml xv xvid zeroconf zlib" ABI_X86="32" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="synaptics libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

sys-apps/kexec-tools-2.0.22::gentoo was built with the following:
USE="lzma zlib -booke -xen"

sublogic · Post by **sublogic** » Thu Mar 24, 2022 7:06 pm

Ah. It's a big initramfs (genkernel) and the capture kernel needs more reserved memory. in my case: "crashkernel=172M nokaslr".

Found by bisection. After the test panic, the capture kernel kicks in and enters early userspace no problem. I can see the /proc/vmcore from the rescue shell.

Alas, there is not enough memory left to open my LUKS2 volume

. (This operation is a memory hog; the cryptsetup FAQ explains why.)

Okay, not quite solved, but I have options.

Shrink the initramfs; I don't need its /lib/modules.
Copy /proc/vmcore to a USB stick from early userspace.

sublogic · Post by **sublogic** » Sun Apr 17, 2022 3:44 am

Found a better way than the hack to stay on the 80x25 VGA: in my case: compile radeon built-in rather than as a module, and add the required firmware to the kernel.

To know what module takes over your console,

Code: Select all

$ dmesg | egrep -i 'console|fbcon'
[    0.306475] Console: colour dummy device 80x25
[    0.307525] printk: console [tty0] enabled
[    3.343837] fbcon: Taking over console
[    3.687242] Console: switching to colour frame buffer device 128x48
[   45.822346] Console: switching to colour dummy device 80x25
[   45.822477] radeon 0000:01:05.0: vgaarb: deactivate vga console
[   46.633315] fbcon: radeondrmfb (fb0) is primary device
[   46.665027] Console: switching to colour frame buffer device 160x50

So it's the radeon module that takes over at 45.8 seconds.

To know what firmware is needed,

Code: Select all

$ dmesg | grep firmware
[   44.097441] Loading firmware: regulatory.db
[   44.608538] Loading firmware: regulatory.db.p7s
[   45.829344] Loading firmware: radeon/R300_cp.bin

The regulatory.db is wireless stuff. The graphics firmware I need is radeon/R300_cp.bin. So, in my kernel config,

Code: Select all

CONFIG_EXTRA_FIRMWARE="radeon/R300_cp.bin"
CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"

If you need more than one firmware file, use a space-separated list in CONFIG_EXTRA_FIRMWARE.

Result with the new kernel:

Code: Select all

$ dmesg | egrep -i 'console|fbcon'
[    0.316941] Console: colour dummy device 80x25
[    0.317961] printk: console [tty0] enabled
[    3.358602] fbcon: Taking over console
[    3.723760] radeon 0000:01:05.0: vgaarb: deactivate vga console
[    3.801895] fbcon: radeondrmfb (fb0) is primary device
[    3.833340] Console: switching to colour frame buffer device 160x50

Note the timestamps. The switch happens much earlier, in time for the early userspace if you use an initramfs.

With this change the crash recovery kernel has no trouble initializing the video. No need for --reset-vga or --console-vga.
Drawback: you have to rebuild your kernel when sys-kernel/linux-firmware is updated. Or not, just wait for the next kernel.
Advantage: you can capture crash dumps triggered by programs that run under X.

For the out-of-memory condition when decrypting the LUKS volume, add a temporary passphrase using the pbkdf2 key derivation function.

Code: Select all

# cryptsetup luksAddKey --key-slot X --pbkdf pbkdf2 /dev/sdaY

X is some unused key slot, sdaY is your LUKS partition. Unlike the default argon2, the pbkdf2 doesn't need large memory to process the passphrase. Assuming a genkernel initramfs, boot with "debug" and open the LUKS volume from the rescue shell.

Code: Select all

cryptsetup luksOpen --key-slot X /dev/sdaY root

After you exit the rescue shell the /init script finds the volume already open and continues from there.

(Edit 2023-04-03: use --pbkdf with cryptsetup rather than the old --luks1 option.)