Hibernation PM Image not found on resume - LUKS swap

[oq2pi]

I am running an AMD64 system that is otherwise working, suspends to RAM fine, but at hibernation resume time, issuing what I think is the correct <major>:<minor> into /sys/power/resume the kernel gives the error:

Code: Select all

PM: Image not found (code -22)

The system is configured with a stable gentoo-sources kernel, OpenRC, and a minimal custom initramfs. I do not use genkernel. The custom initramfs, built with my own script based on Gentoo wiki instructions, is able to boot the system with no issues other than this one with hibernation. The initramfs kernel matches the kernel I am booting. It is using a swap partition, not a swap file. The swap partition is LUKS encrypted.

The kernel is built with what I think are all the appropriate options. .config snippet:

Code: Select all

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not set
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_HIBERNATION_SNAPSHOT_DEV=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_PM_SLEEP_DEBUG=y
CONFIG_DPM_WATCHDOG=y
CONFIG_DPM_WATCHDOG_TIMEOUT=120
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_CLK=y

The kernel commandline contains:

Code: Select all

resume=UUID=8bc99d4c-1f6f-4gaf-8bfb-b68aa7731408

I have verified multiple times that this is the correct UUID of the inner swap volume, e.g., if after cryptsetup luksOpen the swap volume is located at /dev/mapper/luks_swap , the UUID given is that of /dev/mapper/luks_swap , not the enclosing LUKS volume.

The initramfs init script is able to successfully luksOpen the swap volume. The relevant snip from the init script is:

Code: Select all

cryptsetup luksOpen --key-file=/swap.key $(findfs UUID=15760a09-19bc-4951-8c23-9df6c07ddf15) luks_swap || rescue_shell
echo "Debugging hibernation resume.  Enter r or R for rescue shell, anything else to continue with initscript"
read dbg_input
case $dbg_input in
  [Rr] )
    rescue_shell
    ;;
  *)
   echo "  Continuing...";;
esac
printf '%u:%u\n' $(stat -L -c '0x%t 0x%T' /dev/mapper/luks_swap) > /sys/power/resume || rescue_shell

If I enter the rescue shell and poke around, I can confirm the swap volume is has properly completed cryptsetup luksOpen, it is mapped in /dev/mapper at the intended name, the stat command identifies the major and minor device numbers, and the printf command properly generates the decimal values that /sys/power/resume should expect.

Looking at kernel output via dmesg, I get the "PM: Image not found (code -22)" error as noted above. If the initscript is allowed to continue, the system boots properly as it would on a clean boot, no special errors (other than if it was hibernated, I see the the filesystem consistency issues one expects when booting cold when the system had been hibernated).

When the system is booted, the swap volume is properly setup in /etc/conf.d/dmcrypt , has a proper entry in /etc/fstab:

Code: Select all

/dev/mapper/luks_swap          none            swap            sw                                                              2

swapon --show shows the swap is activated as expected:

Code: Select all

NAME                  TYPE       SIZE USED PRIO
# swapon --show
/dev/mapper/luks_swap partition 32.2G   0B   -2

The swap partition is larger than the amount of RAM on the system, and as you see, when I try it, there is no swap actually being used.

blkid confirms the UUID of the inner swap volume is the same as used in the kernel command-line:

Code: Select all

# blkid --probe /dev/mapper/luks_swap
/dev/mapper/luks_swap: LABEL="swappart" UUID="8bc99d4c-1f6f-4gaf-8bfb-b68aa7731408" VERSION="1" TYPE="swap" USAGE="other"

When I attempt to hibernate, using loginctl hibernate , it looks like it is doing what I would expect. Screen blanks, takes a bit to write the hibernation image, power LED is flashing as it does so, screen briefly flashes up, then power goes off.

Help is requested to figure out:

• Is the image not being written to the right place or in the right format?
• Is something not being properly done in the resume attempt?
• Is something wrong with kernel or drivers that would cause the image to be corrupted somehow so that the kernel resume doesn't see it as valid?
• Something else that I'm missing??

[sublogic]

Code: Select all

printf '%u:%u\n' $(stat -L -c '0x%t 0x%T' /dev/mapper/luks_swap) > /sys/power/resume || rescue_shell

This can be simplified to:

Code: Select all

stat -L -c '%Hr:%Lr' /dev/mapper/luks_swap > /sys/power/resume || rescue_shell

but that doesn't address the issue. You seem to be doing everything right. Are the correct major:minor numbers still in /sys/power/resume after a failed resume ?

Can you add hexdump and head to your initramfs ? Then, in the rescue shell, you can check manually for a hibernation signature after the luksOpen.
On my hibernated laptop, from the rescue shell,

Code: Select all

# hexdump -C /dev/dm-1 | head

I see SWAPSPACE2S1SUSPEND at offset 0xfec(=4076 decimal). The S1SUSPEND is the signature (see kernel/power/swap.c in the kernel sources).
(My swap is /dev/dm-1 and not anything friendlier because udev isn't running during this exercise. But I digress.)

If the S1SUSPEND is not there, you have to find out where the hibernation image went...

(Uh, where did the /swap.key come from ? Hard-coded in the initramfs ? That's not very secure, you know...)

[oq2pi]

Hi sublogic, thanks for the help!

Code: Select all

printf '%u:%u\n' $(stat -L -c '0x%t 0x%T' /dev/mapper/luks_swap) > /sys/power/resume || rescue_shell

This can be simplified to:

Code: Select all

stat -L -c '%Hr:%Lr' /dev/mapper/luks_swap > /sys/power/resume || rescue_shell

I pulled this line from viewtopic-t-1146363-highlight-custom+in ... +luks.html but I tried what you posted and realized why @mvasi90 did it that way. BusyBox's stat doesn't support the %Hr and %Lr format specifiers. It's more bare-bones than the coreutils stat (which I did not pull into the initramfs).

Are the correct major:minor numbers still in /sys/power/resume after a failed resume ?

Yes, they do stay there after a failed resume.

Can you add hexdump and head to your initramfs ? Then, in the rescue shell, you can check manually for a hibernation signature after the luksOpen.
On my hibernated laptop, from the rescue shell,

Code: Select all

# hexdump -C /dev/dm-1 | head

I see SWAPSPACE2S1SUSPEND at offset 0xfec(=4076 decimal). The S1SUSPEND is the signature (see kernel/power/swap.c in the kernel sources).

Some good insight -- I pulled both hexdump and head into the initramfs and tried it. Just as when I tried this before the hibernate, the S1SUSPEND was not there. If the hibernate header isn't present, that would explain why the PM: Image not found error occurs. Recommendations on how to figure out where this hibernation image is actually being written? I don't see any verbose options in loginctl to tell the hibernation destination...

(Uh, where did the /swap.key come from ? Hard-coded in the initramfs ? That's not very secure, you know...)

I don't follow. The initramfs is on a LUKS-encrypted standalone /boot volume. GRUB asks for the /boot passphrase at boot time. Once the encrypted /boot is unlocked, the initramfs does enough unlocking so that none of the other volumes prompt for passwords. If I can get / mounted just by initiating the boot, how is the initramfs being able to unlock swap less secure than the standard sequence of boot events being able to unlock the rest of the system? I could have the boot process request secondary passphrases, but I don't think most people use multiple layers of passphrases?

[sublogic]

Look in /var/log/messages and /var/log/dmesg for hibernation errors left there by the suspending kernel.

Go through the steps in /usr/src/linux/Documentation/power/basic-pm-debugging.rst.

Not all drivers support hibernation. If built as modules, you can unload them before hibernating and modprobe them back after resuming. sys-power/hibernate-script does this and has a blocklist of problematic modules; emerge it and just run sudo hibernate behind elogind's back. If it fails, depclean it and move on. If it works, you can plant scripts in /lib64/elogind/system-sleep to take care of the bad modules. Read the loginctl manpage for details.

The initramfs is on a LUKS-encrypted standalone /boot volume.

Ah, that explains it. You're safe then. I hope you find a solution.

[oq2pi]

Thanks for the recommendations on what to do next. It's going to be a few days before I can set some time aside to try the debugging steps you mentioned, but looking at the kernel syslog, I see the following which corresponds to when I invoked the hibernate command:

Code: Select all

Feb 04 19:59:36 [kernel] [  339.912399] PM: hibernation: hibernation entry
Feb 04 19:59:36 [kernel] [  339.912501] Loading firmware: iwlwifi-8265-36.ucode
Feb 04 19:59:36 [kernel] [  339.913372] Loading firmware: rtl_nic/rtl8168g-3.fw
Feb 04 19:59:36 [kernel] [  339.913381] Loading firmware: amdgpu/raven_gpu_info.bin
Feb 04 19:59:36 [kernel] [  339.913387] Loading firmware: amdgpu/raven_sdma.bin
Feb 04 19:59:36 [kernel] [  339.913391] Loading firmware: amdgpu/raven_asd.bin
Feb 04 19:59:36 [kernel] [  339.913396] Loading firmware: amdgpu/raven_ta.bin
Feb 04 19:59:36 [kernel] [  339.913399] Loading firmware: amdgpu/raven_pfp.bin
Feb 04 19:59:36 [kernel] [  339.913404] Loading firmware: amdgpu/raven_me.bin
Feb 04 19:59:36 [kernel] [  339.913408] Loading firmware: amdgpu/raven_ce.bin
Feb 04 19:59:36 [kernel] [  339.913413] Loading firmware: amdgpu/raven_rlc.bin
Feb 04 19:59:36 [kernel] [  339.913417] Loading firmware: amdgpu/raven_mec.bin
Feb 04 19:59:36 [kernel] [  339.913422] Loading firmware: amdgpu/raven_mec2.bin
Feb 04 19:59:36 [kernel] [  339.913425] Loading firmware: amdgpu/raven_vcn.bin
Feb 04 19:59:36 [kernel] [  339.913431] Loading firmware: rtl_nic/rtl8153b-2.fw

These are the last lines before the log entries correspond to the following boot.

Is it normal right after hibernation entry to then see it loading firmware?