Kernel panics on boot due to misconfigured LUKS setup.

[leyvi]

Well, it almost got there...

First try there were some issues with the HID modules, so I built those into the kernel (the ones that were giving error messages) and the errors went away.
Everything looks fine, that is, it prompts me for the disk's passphrase, but after I type it and press <Enter>, nothing happens, no matter how long I wait.

What now?
I can't check dmesg since I doubt that the root filesystem got mounted...

which HID modules did you need to build in?

The prompt looks like a cryptsetup prompt right? Do you have any args like "quiet" added to your kernel command line?

It sounds like cryptsetup is hanging which is abnormal. Generally if it's missing parts it'll log failure info to the console. There is a small chance it's running out of memory running the KDF but it will also typically warn about this and not just stall.

amd_sfh and its dependencies (the HID module itself).

Command line looks like this:

Code: Select all

resume=/swap/swapfile hugepagesz=1GB hugepages=8 pcie_aspm=force

Maybe the resume, since that filesystem isn't mounted yet?

[zen_desu]

Well, it almost got there...

First try there were some issues with the HID modules, so I built those into the kernel (the ones that were giving error messages) and the errors went away.
Everything looks fine, that is, it prompts me for the disk's passphrase, but after I type it and press <Enter>, nothing happens, no matter how long I wait.

What now?
I can't check dmesg since I doubt that the root filesystem got mounted...

which HID modules did you need to build in?

The prompt looks like a cryptsetup prompt right? Do you have any args like "quiet" added to your kernel command line?

It sounds like cryptsetup is hanging which is abnormal. Generally if it's missing parts it'll log failure info to the console. There is a small chance it's running out of memory running the KDF but it will also typically warn about this and not just stall.

amd_sfh and its dependencies (the HID module itself).

Command line looks like this:

Code: Select all

resume=/swap/swapfile hugepagesz=1GB hugepages=8 pcie_aspm=force

Maybe the resume, since that filesystem isn't mounted yet?

ugrd won't try to handle resumes unless you enable the module for it (ugrd.fs.resume)

When it's asking for the password it's running `cryptsetup open`. I've generally seen it log an error and exit if kmods or userspace stuff is missing, or it OOMs.

You should also see a log line like "Opening cryptsetup device: name" and then a failure or success message after cryptsetup is done. If you don't see the message after, cryptsetup seems to be hanging for some reason

[zen_desu]

Well, it almost got there...

First try there were some issues with the HID modules, so I built those into the kernel (the ones that were giving error messages) and the errors went away.
Everything looks fine, that is, it prompts me for the disk's passphrase, but after I type it and press <Enter>, nothing happens, no matter how long I wait.

What now?
I can't check dmesg since I doubt that the root filesystem got mounted...

which HID modules did you need to build in?

The prompt looks like a cryptsetup prompt right? Do you have any args like "quiet" added to your kernel command line?

It sounds like cryptsetup is hanging which is abnormal. Generally if it's missing parts it'll log failure info to the console. There is a small chance it's running out of memory running the KDF but it will also typically warn about this and not just stall.

amd_sfh and its dependencies (the HID module itself).

Command line looks like this:

Code: Select all

resume=/swap/swapfile hugepagesz=1GB hugepages=8 pcie_aspm=force

Maybe the resume, since that filesystem isn't mounted yet?

ugrd won't try to handle resumes unless you enable the module for it (ugrd.fs.resume)

When it's asking for the password it's running `cryptsetup open`. I've generally seen it log an error and exit if kmods or userspace stuff is missing, or it OOMs.

You should also see a log line like "Opening cryptsetup device: name" and then a failure or success message after cryptsetup is done. If you don't see the message after, cryptsetup seems to be hanging for some reason


If you want to debug this at runtime, you can build ugrd with the "ugrd.base.debug" module and it will give you a bash shell early on. you can try running cryptsetup manually or run "crypt_init" to run the builtin unlock script (the same thing that runs automatically without debug mode). Once you're done you can "exit" and it should continue booting

[leyvi]

Well, adding the µgRD resume module didn't help.

I think something's wrong with the keyboard (on the software side), and typing on it doesn't result in anything happening. It works fine in the LiveCD though.

[pietinger]

I think something's wrong with the keyboard (on the software side), and typing on it doesn't result in anything happening. It works fine in the LiveCD though.

Do you have enabled CONFIG_KEYBOARD_ATKBD as built-in <*> in your kernel?

... and yes, I know this should be only needed for a PS/2 keyboard ... but I have got a report (cannot remember) that this was missing to have a keyboard IN THE INITRAMFS.

[leyvi]

I think something's wrong with the keyboard (on the software side), and typing on it doesn't result in anything happening. It works fine in the LiveCD though.

Do you have enabled CONFIG_KEYBOARD_ATKBD as built-in <*> in your kernel?

... and yes, I know this should be only needed for a PS/2 keyboard ... but I have got a report (cannot remember) that this was missing to have a keyboard IN THE INITRAMFS.

Trying it out now, thanks!

[leyvi]

Didn't work, same thing happened...

Gonna double-check that all of these:

Code: Select all

Cryptographic API  --->
   [*]   Hardware crypto devices  --->
      [*]   Support for AMD Secure Processor
      <*>     Secure Processor device driver
      [*]       Platform Security Processor (PSP) device
Device Drivers  --->
   I2C Support  --->
      <*> I2C support
      <*>   I2C device interface
      I2C Hardware Bus Support  --->
         <*> Intel PIIX4 and compatible (ATI/AMD/Serverworks/Broadcom/SMSC)
         <*> Synopsys DesignWare Platform
         [*]   AMD PSP I2C semaphore support
   [*] Pin controllers  --->
      [*]   AMD GPIO pin control
   [*] HID bus support  --->
      <*>   Generic HID driver
            Special HID drivers  --->
               <*> HID Multitouch panels
            USB HID support  --->
               <*> USB HID transport layer
      <*>   I2C HID support  --->
               <M>   HID over I2C transport layer ACPI driver

are set correctly, again, just in case.

[pietinger]

Didn't work, same thing happened... [1]

[...] [2] are set correctly, again, just in case.

1. This is strange ...because for a keyboard usually you will need only CONFIG_INPUT_EVDEV (and in some cases (I dont know which) the KEYBOARD_ATKBD).
2. This is for a mouse/touchpad ... not for a keyboard.

So, you said you can use the keyboard with our dist-kernel ... so, maybe check all differences ... (sometimes this is the only way to find out what is missing ... see in this thread: https://forums.gentoo.org/viewtopic-t-1 ... ight-.html )

... and yes, amd_sfh should really NOT be necessary to have a working keyboard.

[zen_desu]

... and yes, amd_sfh should really NOT be necessary to have a working keyboard.

yeah I found this surprising. ugrd does _basic_ checks for input drivers (reads devices from /sys/class/input and adds the driver for them if they report having 25+ keys)

https://github.com/desultory/ugrd/blob/ ... py#L31-L43

generally this will find the correct kmod unless the system requires multiple subsystems for the keyboard (sometimes the case on Apple stuff)

using a usb keyboard may be a decent test, those almost always use simple/common drivers.

[leyvi]

It works now!

You'll never guess what the problem was...

Gonna double-check that all of these: ... are set correctly, again, just in case.

There were a whole bunch of modules on that list from the wiki page on the Framework 16 that were set to m instead of y, changing that solved the issue completely.

[zen_desu]

It works now!

You'll never guess what the problem was...

Gonna double-check that all of these: ... are set correctly, again, just in case.

There were a whole bunch of modules on that list from the wiki page on the Framework 16 that were set to m instead of y, changing that solved the issue completely.

which ones in particular? theoretically this should be something an initramfs can look for before booting, but in cases where "low level" stuff is provided by a vendor specific module, that becomes hard/impossible (especially when builtin because then the kernel doesn't report a device using that module directly)

[leyvi]

It works now!

You'll never guess what the problem was...

Gonna double-check that all of these: ... are set correctly, again, just in case.

There were a whole bunch of modules on that list from the wiki page on the Framework 16 that were set to m instead of y, changing that solved the issue completely.

which ones in particular? theoretically this should be something an initramfs can look for before booting, but in cases where "low level" stuff is provided by a vendor specific module, that becomes hard/impossible (especially when builtin because then the kernel doesn't report a device using that module directly)

Most, but not all, of these:

Code: Select all

 Device Drivers  --->
   I2C Support  --->
      <*> I2C support
      <*>   I2C device interface
      I2C Hardware Bus Support  --->
         <*> Intel PIIX4 and compatible (ATI/AMD/Serverworks/Broadcom/SMSC)
         <*> Synopsys DesignWare Platform
         [*]   AMD PSP I2C semaphore support
   [*] Pin controllers  --->
      [*]   AMD GPIO pin control
   [*] HID bus support  --->
      <*>   Generic HID driver
            Special HID drivers  --->
               <*> HID Multitouch panels
            USB HID support  --->
               <*> USB HID transport layer
      <*>   I2C HID support  --->
               <M>   HID over I2C transport layer ACPI driver

[zen_desu]

It works now!

You'll never guess what the problem was...There were a whole bunch of modules on that list from the wiki page on the Framework 16 that were set to m instead of y, changing that solved the issue completely.

which ones in particular? theoretically this should be something an initramfs can look for before booting, but in cases where "low level" stuff is provided by a vendor specific module, that becomes hard/impossible (especially when builtin because then the kernel doesn't report a device using that module directly)

Most, but not all, of these:

Code: Select all

 Device Drivers  --->
   I2C Support  --->
      <*> I2C support
      <*>   I2C device interface
      I2C Hardware Bus Support  --->
         <*> Intel PIIX4 and compatible (ATI/AMD/Serverworks/Broadcom/SMSC)
         <*> Synopsys DesignWare Platform
         [*]   AMD PSP I2C semaphore support
   [*] Pin controllers  --->
      [*]   AMD GPIO pin control
   [*] HID bus support  --->
      <*>   Generic HID driver
            Special HID drivers  --->
               <*> HID Multitouch panels
            USB HID support  --->
               <*> USB HID transport layer
      <*>   I2C HID support  --->
               <M>   HID over I2C transport layer ACPI driver

Yeah, seems your keyboard is likely i2c and that won't function (even with the hid drivers) unless the i2c "backend" is also initialized.

I've been looking for ways to reliably find the module in use for this stuff, but it's often builtin so you can't truly "resolve" the driver to a particular module.

[sublogic]

Glad to see that you can boot now, but I wouldn't try to hibernate if I were you:

Command line looks like this:

Code: Select all

resume=/swap/swapfile hugepagesz=1GB hugepages=8 pcie_aspm=force

Maybe the resume, since that filesystem isn't mounted yet?

(Emphasis mine.)
You use a swap file instead of a partition ? (It looks like your LUKS volume has the root filesystem and nothing else, no LVM and no partitions.) Resuming from a swap file is not so simple.

Here are some excerpts from files in /usr/src/linux/Documentation/power/.

Code: Select all

   **BIG FAT WARNING**

   If you touch anything on disk between suspend and resume...
                                ...kiss your data goodbye.

   If you do resume from initrd after your filesystems are mounted...
                                ...bye bye root partition.

                        [this is actually same case as above]

so it's a good thing that the filesystem isn't mounted yet.

Code: Select all

[...]
    resume=<swap_file_partition> resume_offset=<swap_file_offset>

so you don't mount, instead you give a partition and an offset; but that doesn't apply because

Code: Select all

[...]
Use a userland suspend application that will set the partition and offset
with the help of the SNAPSHOT_SET_SWAP_AREA ioctl described in
Documentation/power/userland-swsusp.rst (this is the only method to suspend
to a swap file allowing the resume to be initiated from an initrd or initramfs
image).

The file userland-swsusp.rst, which I won't quote here, describes how to write a userland suspend application. Fun reading.

sys-power/suspend seems to be such a userland suspend application (don't know, I don't use it) but its goals are different. I don't see any mention of SNAPSHOT_SET_SWAP_AREA.

So your options are:

1. Don't hibernate.
2. Scour the Internet for a userland suspend application that supports resuming from a swap file.
3. Install sys-power/suspend, study it and hack it to add swap file functionality.
4. Remodel your /dev/mapper/rootfs. Backup your install; add LVM, or just create a root and a swap partition without LVM; restore your install.

(I'm pretty sure you can partition the /dev/mapper/rootfs directly, like other block devices. @zen_desu: would ugrd detect such a setup ?)

EDIT: the Archwiki implies that your initramfs can echo the file offset to /sys/power/resume_offset and then the major:minor to /sys/power/resume ? But that contradicts the statement in swsusp-and-swap-files.rst .

[zen_desu]

Code: Select all

   **BIG FAT WARNING**

   If you touch anything on disk between suspend and resume...
                                ...kiss your data goodbye.

   If you do resume from initrd after your filesystems are mounted...
                                ...bye bye root partition.

                        [this is actually same case as above]

so it's a good thing that the filesystem isn't mounted yet.

Code: Select all

[...]
    resume=<swap_file_partition> resume_offset=<swap_file_offset>

so you don't mount, instead you give a partition and an offset; but that doesn't apply because

This big fat warning is exactly why I'm extremely cautious with resume support in ugrd

@zen_desu: would ugrd detect such a setup ?)

it does not detect it, users must manually enable the "ugrd.fs.resume" for it to attempt to handle "resume=" from the kernel command line.

the main branch _does not_ currently support encrypted resume because as you shared, doing it and accidentally writing can be catastrophic. Another user made a branch which handles resuming from a swap partition under luks/lvm or a swapfile on an encrypted volume, but I haven't had the time to give it a super thorough review and add extensive testing (I want to try to add test cases where a write to the volume is attempted and prevented)

the current test branch for it seems fine to me, but I'm exercising a lot of caution here mostly because I don't do hibernation myself, so I would probably not be able to personally "battle test" this like I test most of the other features which are "hard to test" such as yubikey support.

my personal stance on hibernation is more or less: suspend to ram + screen lock is fine/preferable
on an encrypted setup, I don't want my RAM contents to be stored anywhere when my system is off (even if encrypted), and don't feel comfortable assuming the risk involved with resuming "improperly"

[leyvi]

My root filesystem is btrfs, and the swapfile is on its own subvolume, as described on the ArchWiki page about the filesystem. It has several special options enabled, so it's not just a swap file (I think). Does that make things any easier?

[leyvi]

I went back to the wiki to read more:
Apparently, there's a UEFI variable called HibernateLocation used by systemd to keep track of the swap.
zen_desu, maybe you could integrate that with ugRD somehow? btrfs would require some special care.

[leyvi]

Here's what I'm thinking:
On shutdown (using the local runlevel, I think), determine if hibernating using sysfs, and if so, get the device and optionally offset of swap (btrfs requires something special). Put this information in the aforementioned HibernateLocation EFI variable, and if not hibernating, clear the variable. In early userspace, check if the variable is set, and if so, use it to resume, assuming the partition is available. This will be tricky to handle, since the partition may need to be decrypted first.

About /sys/power/{resume,resume_offset}:
The Arch wiki suggests that these are persistent; it seems to say that you may echo the desired values into them, and it is like kernel arguments for the next boot? That doesn't make a lot of sense to me…

When is resume data read from swap anyway? Early userspace?

[zen_desu]

I went back to the wiki to read more:
Apparently, there's a UEFI variable called HibernateLocation used by systemd to keep track of the swap.
zen_desu, maybe you could integrate that with ugRD somehow? btrfs would require some special care.

Do you know where this uefi var is stored? just plain efivars?

I could look into that, but currently (when the ugrd.fs.resume module is activated) it will try to parse "resume=" from the kernel command line which should be good enough.

There is a branch with more advanced resume code which should allow resuming from a device or file under a fs (right now it only looks for devices early in the boot process to avoid resuming after writes could have been made). The newer version has some more checks and should be able to handle most storage types, whether they be under LVM, LUKS, or a file under a particular filesystem.

the PR is here and I think the user who made it has been using it safely: https://github.com/desultory/ugrd/pull/290

[zen_desu]

Here's what I'm thinking:
On shutdown (using the local runlevel, I think), determine if hibernating using sysfs, and if so, get the device and optionally offset of swap (btrfs requires something special). Put this information in the aforementioned HibernateLocation EFI variable, and if not hibernating, clear the variable. In early userspace, check if the variable is set, and if so, use it to resume, assuming the partition is available. This will be tricky to handle, since the partition may need to be decrypted first.

About /sys/power/{resume,resume_offset}:
The Arch wiki suggests that these are persistent; it seems to say that you may echo the desired values into them, and it is like kernel arguments for the next boot? That doesn't make a lot of sense to me…

When is resume data read from swap anyway? Early userspace?

Yes, the resume process happens in the initramfs. The process is started by writing info about the device/file being resume from into a /sys/power/resume. At this point the system should resume back into the state it was in when you hibernated.

[leyvi]

I went back to the wiki to read more:
Apparently, there's a UEFI variable called HibernateLocation used by systemd to keep track of the swap.
zen_desu, maybe you could integrate that with ugRD somehow? btrfs would require some special care.

Do you know where this uefi var is stored? just plain efivars?

I could look into that, but currently (when the ugrd.fs.resume module is activated) it will try to parse "resume=" from the kernel command line which should be good enough.

There is a branch with more advanced resume code which should allow resuming from a device or file under a fs (right now it only looks for devices early in the boot process to avoid resuming after writes could have been made). The newer version has some more checks and should be able to handle most storage types, whether they be under LVM, LUKS, or a file under a particular filesystem.

the PR is here and I think the user who made it has been using it safely: https://github.com/desultory/ugrd/pull/290

I don't know where this variable is, it looks like regular efivarfs. I'll check though.

[leyvi]

Here's what I'm thinking:
On shutdown (using the local runlevel, I think), determine if hibernating using sysfs, and if so, get the device and optionally offset of swap (btrfs requires something special). Put this information in the aforementioned HibernateLocation EFI variable, and if not hibernating, clear the variable. In early userspace, check if the variable is set, and if so, use it to resume, assuming the partition is available. This will be tricky to handle, since the partition may need to be decrypted first.

About /sys/power/{resume,resume_offset}:
The Arch wiki suggests that these are persistent; it seems to say that you may echo the desired values into them, and it is like kernel arguments for the next boot? That doesn't make a lot of sense to me…

When is resume data read from swap anyway? Early userspace?

Yes, the resume process happens in the initramfs. The process is started by writing info about the device/file being resume from into a /sys/power/resume. At this point the system should resume back into the state it was in when you hibernated.

Then you should just be able to do the following:

1. On shutdown, determine if entering hibernation; if so, set the EFI variable. If not, clear it.
2. On boot, in early userspace, check the EFI variable, and if not empty, parse it and toss the result into the sysfs handles for resume (and optionally resume_offset, in the case of a swap file).

This should be pretty simple, and it could probably be done with two simple shell-scripts. I'll take a look if I find some time, maybe in a week or two, unless it gets done first. Again, it's pretty simple in theory…

[leyvi]

About the EFI variable mentioned on the Arch wiki: I was unable to find it in efivarfs.