ugrd, luks2, lvm, encrypted root (solved)

[mikb]

For a number of years, I have used genkernel to install kernels and create the appropriate initrd. I had it successfully unlocking the LUKS2 partition on the SSD using a keyfile overlaid in the initramfs (any comments as to the pros and cons of this will receive a rude answer - I know what I'm doing), activating the volume group, and mounting the volumes and subvolumes (btrfs) successfully.

I have a new machine now with dual SSDs, and I attempting to reconstruct all of this with the volume group spread across the two SSDs.

Since genkernel is apparently deprecated. I'm trying to migrate initrd generation to ugRD.

The generated initrd isn't working, and the documentation is a bit sparse on the stuff I'm trying to do.

It looks like the init is failing to mount the non-root volumes and subvolumes, which suggests to me that the LVM configuration has failed. the debug info claims to have mounted rootfs on /, but I suspect it's confused.

Details in subsequent posts.

Does any one have any idea what's broken?

[mikb]

LABEL=ROOT / btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@root 1 1
LABEL=ROOT /usr btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@usr 1 2
LABEL=ROOT /usr/local btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@local 1 2
LABEL=ROOT /opt btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@opt 1 2
LABEL=VAR /var btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@var 1 3
LABEL=HOME /home btrfs compress=zstd,noatime,space_cache=v2,ssd,discard=async,subvol=@home 1 3
LABEL=SWAP none swap sw

[mikb]

Code: Select all

modules = [
 "ugrd.crypto.cryptsetup",
 "ugrd.kmod.nvme",
 "ugrd.fs.btrfs",
 "ugrd.fs.lvm",
 "ugrd.fs.resume",
]

kmod_ignore_video = false

out_dir = "/boot"

cpio_compression = "zstd"  # Use zstd compression

kmod_autodetect_lspci = true

paths = ["/root/keys"]

autodetect_root_dm = false
autodetect_root_lvm = true
autodetect_root_luks = true


[mounts.usr]
options = ["compress=zstd","noatime","space_cache=v2","ssd","discard=async","subvol=@usr"]
type = "btrfs"
destination = "/usr"
label = "ROOT"
no_validate = true

[mounts.local]
options = ["compress=zstd","noatime","space_cache=v2","ssd","discard=async","subvol=@local"]
type = "btrfs"
destination = "/usr/local"
label = "ROOT"
no_validate = true

[mounts.opt]
options = ["compress=zstd","noatime","space_cache=v2","ssd","discard=async","subvol=@opt"]
type = "btrfs"
destination = "/opt"
label = "ROOT"
no_validate = true

[mounts.var]
options = ["compress=zstd","noatime","space_cache=v2","ssd","discard=async","subvol=@var"]
type = "btrfs"
destination = "/var"
label = "VAR"
no_validate = true

[mounts.home]
options = ["compress=zstd","noatime","space_cache=v2","ssd","discard=async","subvol=@home"]
type = "btrfs"
destination = "/home"
label = "HOME"
no_validate = true

[cryptsetup.ROOT0]
uuid = "816afe48-38e7-4b3e-9c55-68ec9729f157"
key_file = "/root/keys/ROOT"
include_key = true

[cryptsetup.ROOT1]
uuid = "ee0a6029-e5d3-4566-9201-ad2f39ec8318"
key_file = "/root/keys/ROOT"
include_key = true

[lvm.TUXEDO_IB14Pro_VG00]
uuid = "QICb9J-VtcP-q6fo-1FqW-u0Ro-dksb-ZzCMG2"
holders = "ROOT"

[mikb]

Code: Select all

         NAME                                    TYPE  LABEL UUID
         nvme0n1                                 disk         
         ├─nvme0n1p1                             part        B667-03E9
         ├─nvme0n1p2                             part        86b946e2-9da1-4145-8efa-29beb10579f8
         │ └─BOOT0                               crypt BOOT  55a1d8cd-9cbd-4b7a-a32f-fce4024f9a0c
         └─nvme0n1p3                             part        ee0a6029-e5d3-4566-9201-ad2f39ec8318
           └─ROOT0                               crypt       eFOmyT-eUOg-0A1i-ogt4-GKaS-nyOi-CG99kG
             ├─TUXEDO_IB14Pro_VG00-SWAP0         lvm         229955c3-859a-414d-80c0-d197f2338ea6
     ┌┈▶     ├─TUXEDO_IB14Pro_VG00-ROOT_rmeta_0  lvm          
     ├┈▶     ├─TUXEDO_IB14Pro_VG00-ROOT_rimage_0 lvm          
  ┌┈▶┆       ├─TUXEDO_IB14Pro_VG00-VAR_rmeta_0   lvm          
  ├┈▶┆       ├─TUXEDO_IB14Pro_VG00-VAR_rimage_0  lvm          
┌┈▶┆  ┆       ├─TUXEDO_IB14Pro_VG00-HOME_rmeta_0  lvm          
├┈▶┆  ┆       └─TUXEDO_IB14Pro_VG00-HOME_rimage_0 lvm          
┆  ┆  ┆   nvme1n1                                 disk         
┆  ┆  ┆   ├─nvme1n1p1                             part  ESP   D2D7-73F7
┆  ┆  ┆   ├─nvme1n1p2                             part        337a16e8-5ba2-45e2-a7d4-24a47daba8ef
┆  ┆  ┆   │ └─BOOT1                               crypt BOOT  55a1d8cd-9cbd-4b7a-a32f-fce4024f9a0c
┆  ┆  ┆   └─nvme1n1p3                             part        816afe48-38e7-4b3e-9c55-68ec9729f157
┆  ┆  ┆     └─ROOT1                               crypt       bGkhr4-RBK7-mHf3-2GTa-A7ky-xBPx-gVlAxN
┆  ┆  ┆       ├─TUXEDO_IB14Pro_VG00-SWAP1         lvm         acfdf052-eb65-4f17-8892-439e2c50ce5e
┆  ┆  ├┈▶     ├─TUXEDO_IB14Pro_VG00-ROOT_rmeta_1  lvm          
┆  ┆  └┬▶     ├─TUXEDO_IB14Pro_VG00-ROOT_rimage_1 lvm          
┆  ├┈▶ ┆      ├─TUXEDO_IB14Pro_VG00-VAR_rmeta_1   lvm          
┆  └┬▶ ┆      ├─TUXEDO_IB14Pro_VG00-VAR_rimage_1  lvm          
├┈▶ ┆  ┆      ├─TUXEDO_IB14Pro_VG00-HOME_rmeta_1  lvm          
└┬▶ ┆  ┆      └─TUXEDO_IB14Pro_VG00-HOME_rimage_1 lvm          
┆  ┆  └┈┈TUXEDO_IB14Pro_VG00-ROOT                lvm   ROOT  9056a4ad-e011-4280-9189-c80426eacb33
┆  └┈┈┈┈┈TUXEDO_IB14Pro_VG00-VAR                 lvm   VAR   f5fabcf0-3a7f-4ea3-9af8-21b8ca2c1411
└┈┈┈┈┈┈┈┈TUXEDO_IB14Pro_VG00-HOME                lvm   HOME  5c40bd3c-6c24-4fca-a423-6bf545d454da

[mikb]

Code: Select all

# UGRD Filesystem module v7.1.3
LABEL=ROOT                                  /usr                    btrfs           compress=zstd,space_cache=v2,ssd,noatime,discard=async,subvol=@usr
LABEL=ROOT                                  /usr/local              btrfs           subvol=@local,compress=zstd,space_cache=v2,ssd,noatime,discard=async
LABEL=ROOT                                  /opt                    btrfs           compress=zstd,space_cache=v2,ssd,noatime,discard=async,subvol=@opt
LABEL=VAR                                   /var                    btrfs           subvol=@var,compress=zstd,space_cache=v2,ssd,noatime,discard=async
LABEL=HOME                                  /home                   btrfs           compress=zstd,space_cache=v2,ssd,noatime,discard=async,subvol=@home

[zen_desu]

did you try letting ugrd detect it all? (i mean more or less use the default config and add your LUKS volumes if they require keys, etc.) The autodetection should pick up most filesystem layouts and will work with LUKS as long as you don't use key files or have detached headers. If no key file is provided, it will only use plain "cryptsetup open" to let you provide the password while unlocking.

in ugrd, the "mounts" are mostly for things done before the root is decrypted. If you want to force ugrd to mount something which will be carried over with switch_root, you can use late mounts. I would not suggest this as it should automatically detect necessary mounts (like usr and etc) for you. The rest of the mounts (like your home mount) can be mounted by the real init.

[mikb]

did you try letting ugrd detect it all? (i mean more or less use the default config and add your LUKS volumes if they require keys, etc.) The autodetection should pick up most filesystem layouts and will work with LUKS as long as you don't use key files or have detached headers. If no key file is provided, it will only use plain "cryptsetup open" to let you provide the password while unlocking.

That seemed like a good suggestion. Unfortunately, when I tried it, I get unhandled Python KeyError exceptions. The only way I can get the build to complete is to force

Code: Select all

hostonly = false

in the config, which sends me back to having to define everything.

[zen_desu]

did you try letting ugrd detect it all? (i mean more or less use the default config and add your LUKS volumes if they require keys, etc.) The autodetection should pick up most filesystem layouts and will work with LUKS as long as you don't use key files or have detached headers. If no key file is provided, it will only use plain "cryptsetup open" to let you provide the password while unlocking.

That seemed like a good suggestion. Unfortunately, when I tried it, I get unhandled Python KeyError exceptions. The only way I can get the build to complete is to force

Code: Select all

hostonly = false

in the config, which sends me back to having to define everything.

what error do you get when it's auto detecting stuff? this? https://github.com/desultory/ugrd/issues/349

[mikb]

I discovered that setting

Code: Select all

autodetect_root_dm = false

will allow the build to complete. so I can generate am initrd in the installkernel hook. However, it's not succeeding.

If I use

Code: Select all

ugrd --kver 6.18.8-gentoo-x86_64 --test --test-kernel 6.18.8-gentoo-x86_64

I get a Python ValueError exception saying there is no LUKS configuration. Odd.

If I reboot into the installation, I get messages from grub saying that it is loading the kernel and the initrd, and then it hangs.

When building, ugrd does not automatically detect the presence of the LVM volume group, which is disconcerting, to say the least. I have to create an [lvm] table entry in the config file, before it will include any lvm processing.

To recap the disk arrangement, I have two NVME drives. Each is partitioned (GPT) into 3 partitions.

• p1 is the ESP.
• p2 is an encrypted (LUKS2) btrfs partition, with btrfs raid1 between the two drives.
• p3 is an encrypted (LUKS2) LVM PV, with LVM raid1 between the two drives.

The initrd only needs to deal with the p3 pair. Grub successfully handles the p2 raid pair, opening the encrypted volumes, and loading the main grub configuration and theme. The holy grail is to use keys from the TPM to open the LUKS2 partitions, but for the moment I'm content to type passphrases.

The volume group contains three logical volumes, labelled ROOT, VAR, and HOME. Each volume is mirrored across the two PVs.

Because of the way I do backups (via btrfs snapshots), the root of the file system is actually a subvolume (@root) of the logical volume labelled ROOT.

It looks to me like ugrd is understanding none of this.

On the previous system I had no raid, but the same layout otherwise. I was able to get genkernel to handle this, so surely ugrd should be able to handle it as well?

[zen_desu]

I discovered that setting

Code: Select all

autodetect_root_dm = false

will allow the build to complete. so I can generate am initrd in the installkernel hook. However, it's not succeeding.

If I use

Code: Select all

ugrd --kver 6.18.8-gentoo-x86_64 --test --test-kernel 6.18.8-gentoo-x86_64

I get a Python ValueError exception saying there is no LUKS configuration. Odd.

Yes, something is going wrong with the autodetection, if you disable it, it may work. The test suite doesn't support (automatic configuration) for setups like yours (multiple backing devices for the rootfs), so --test won't work here.

https://github.com/desultory/ugrd/blob/ ... d_key.toml Here's an example of the test config for cryptsetup in ugrd (for automated tests). The test suite also doesn't support setups which require interactivity (typical key entry). There is nothing stopping you from trying the tests with QEMU manually, but you need to be sure to setup the test environment with the same uuids, etc.

If I reboot into the installation, I get messages from grub saying that it is loading the kernel and the initrd, and then it hangs.

When building, ugrd does not automatically detect the presence of the LVM volume group, which is disconcerting, to say the least. I have to create an [lvm] table entry in the config file, before it will include any lvm processing.

Yes, this is part of the autodetect_root_dm stuff which won't function when disabled

To recap the disk arrangement, I have two NVME drives. Each is partitioned (GPT) into 3 partitions.

• p1 is the ESP.
• p2 is an encrypted (LUKS2) btrfs partition, with btrfs raid1 between the two drives.
• p3 is an encrypted (LUKS2) LVM PV, with LVM raid1 between the two drives.

The initrd only needs to deal with the p3 pair. Grub successfully handles the p2 raid pair, opening the encrypted volumes, and loading the main grub configuration and theme. The holy grail is to use keys from the TPM to open the LUKS2 partitions, but for the moment I'm content to type passphrases.

The volume group contains three logical volumes, labelled ROOT, VAR, and HOME. Each volume is mirrored across the two PVs.

Because of the way I do backups (via btrfs snapshots), the root of the file system is actually a subvolume (@root) of the logical volume labelled ROOT.

It looks to me like ugrd is understanding none of this.

On the previous system I had no raid, but the same layout otherwise. I was able to get genkernel to handle this, so surely ugrd should be able to handle it as well?

ugrd and genkernel work entirely differently.

ugrd tries to figure you your setup _before_ booting, and use that info. I'm not sure what the deal is, but for some reason your btrfs devices seem to say that they are ontop of the "rmeta" lvm partitions you have?

I've never seen a layout like yours. I'm not sure why you can't simply use btrfs raid and leave lvm out of this, but if the "Detection failure" stuff can be resolved, it should "just work"

[mikb]

I've been quiet for a while, as it turned out I had to resolve a separate problem with a kernel that wouldn't boot. In the end, I threw out my existing kernel configuration, and started from the one in sys-kernel/gentoo-kernel, and started tuning from there. It's much better now.

I managed to make ugrd work. The one remaining weirdness isn't with ugrd, but impacts on validation. Everythime I rebooted into the livecd environment, and ran cryptsetup on the LVM PVs, the UUIDs of the volumes swapped. That breaks validation in ugrd, so I had to turn it off. So for my setup at least, the autodetection has been more hindrance then help.

Should some other benighted creature stumble down the same path I have followed, here's the configuration I'm now using:

Code: Select all

modules = [
  "ugrd.crypto.cryptsetup",
  "ugrd.fs.btrfs",
  "ugrd.fs.lvm",
  "ugrd.kmod.nvme",
]

tmpdir = "/var/tmp"
autodetect_root_dm = false
root_subvol = "/@root"
init_target = "/sbin/openrc-init"
mount_retries = 3
cpio_compression = "ZSTD"
argon2 = true
validate = false

dependencies = [ "/root/keys/ROOT" ]
paths = [ "/root/keys" ]
kernel_modules = [ "dm_raid", "raid1" ]

[cryptsetup.ROOT0]
uuid = "ee0a6029-e5d3-4566-9201-ad2f39ec8318"
key_file = "/root/keys/ROOT"
try_nokey = true

[cryptsetup.ROOT1]
uuid = "816afe48-38e7-4b3e-9c55-68ec9729f157"
key_file = "/root/keys/ROOT"
try_nokey = true

[mounts.root]
type = "btrfs"
uuid = "9056a4ad-e011-4280-9189-c80426eacb33"

[lvm.TUXEDO_IB14Pro_VG00]
uuid = "QICb9J-VtcP-q6fo-1FqW-u0Ro-dksb-ZzCMG2"

Thank you for all your assistance.

[zen_desu]

I've been quiet for a while, as it turned out I had to resolve a separate problem with a kernel that wouldn't boot. In the end, I threw out my existing kernel configuration, and started from the one in sys-kernel/gentoo-kernel, and started tuning from there. It's much better now.

I managed to make ugrd work. The one remaining weirdness isn't with ugrd, but impacts on validation. Everythime I rebooted into the livecd environment, and ran cryptsetup on the LVM PVs, the UUIDs of the volumes swapped. That breaks validation in ugrd, so I had to turn it off. So for my setup at least, the autodetection has been more hindrance then help.

Should some other benighted creature stumble down the same path I have followed, here's the configuration I'm now using:

Code: Select all

modules = [
  "ugrd.crypto.cryptsetup",
  "ugrd.fs.btrfs",
  "ugrd.fs.lvm",
  "ugrd.kmod.nvme",
]

tmpdir = "/var/tmp"
autodetect_root_dm = false
root_subvol = "/@root"
init_target = "/sbin/openrc-init"
mount_retries = 3
cpio_compression = "ZSTD"
argon2 = true
validate = false

dependencies = [ "/root/keys/ROOT" ]
paths = [ "/root/keys" ]
kernel_modules = [ "dm_raid", "raid1" ]

[cryptsetup.ROOT0]
uuid = "ee0a6029-e5d3-4566-9201-ad2f39ec8318"
key_file = "/root/keys/ROOT"
try_nokey = true

[cryptsetup.ROOT1]
uuid = "816afe48-38e7-4b3e-9c55-68ec9729f157"
key_file = "/root/keys/ROOT"
try_nokey = true

[mounts.root]
type = "btrfs"
uuid = "9056a4ad-e011-4280-9189-c80426eacb33"

[lvm.TUXEDO_IB14Pro_VG00]
uuid = "QICb9J-VtcP-q6fo-1FqW-u0Ro-dksb-ZzCMG2"

Thank you for all your assistance.

I'm glad you got it working, which UUIDs in particular are swapped? are you still using LVM raid or are you trying BTRFS raid?


I wouldn't recommend keeping validation disabled globally. Many options let you disable validation for a particular part, which is safer in general. If validation is failing but things are fine, that is generally a ugrd bug and something I'd like to look into

[mikb]

I'm glad you got it working, which UUIDs in particular are swapped? are you still using LVM raid or are you trying BTRFS raid?

cryptsetup.ROOT0 and cryptsetup.ROOT1 kept swapping UUIDs. It's a system thing, and I have no idea why it happens. But it doesn't really matter, as it's a RAID1 mirror and as long as the system opens both UUIDs, we are in normal operation.

The model is working as originally specified.

• An encrypted boot partition, which is a btrfs raid pair over LUKS. Grub is the only thing that deals with that, and with several patches from the grub-devel mailing list, to provide TPM support, AES hardware acceleration, and Argon2id support, is working fine.
• The main system is two LUKS encrypted partitions, each a LVM PV
• The two PVs have multiple LVs defined. The file system LVs are in RAID1 across the two PVs. There's a swap LV on each PC, which just concatenate.

It works with the way my system does backups, and stops runaway world updates from destabilizing the system by choking disk space (which used to happen to me).
By way of explaining the history, first came LVM to replace multiple physical partitions, then LUKS encryption of the PV, then RAIDing the LVs

I wouldn't recommend keeping validation disabled globally. Many options let you disable validation for a particular part, which is safer in general. If validation is failing but things are fine, that is generally a ugrd bug and something I'd like to look into

That's on the list of to-dos. I wasn't sure if I could extend the cryptsetup.ROOT? stanzas with a validate=false, which would allow me to allow validate globally again. If I can, so much the better.

[zen_desu]

I'm glad you got it working, which UUIDs in particular are swapped? are you still using LVM raid or are you trying BTRFS raid?

cryptsetup.ROOT0 and cryptsetup.ROOT1 kept swapping UUIDs. It's a system thing, and I have no idea why it happens. But it doesn't really matter, as it's a RAID1 mirror and as long as the system opens both UUIDs, we are in normal operation.

The model is working as originally specified.

• An encrypted boot partition, which is a btrfs raid pair over LUKS. Grub is the only thing that deals with that, and with several patches from the grub-devel mailing list, to provide TPM support, AES hardware acceleration, and Argon2id support, is working fine.
• The main system is two LUKS encrypted partitions, each a LVM PV
• The two PVs have multiple LVs defined. The file system LVs are in RAID1 across the two PVs. There's a swap LV on each PC, which just concatenate.

It works with the way my system does backups, and stops runaway world updates from destabilizing the system by choking disk space (which used to happen to me).
By way of explaining the history, first came LVM to replace multiple physical partitions, then LUKS encryption of the PV, then RAIDing the LVs

I wouldn't recommend keeping validation disabled globally. Many options let you disable validation for a particular part, which is safer in general. If validation is failing but things are fine, that is generally a ugrd bug and something I'd like to look into

That's on the list of to-dos. I wasn't sure if I could extend the cryptsetup.ROOT? stanzas with a validate=false, which would allow me to allow validate globally again. If I can, so much the better.

Within cryptsetup config definitions, you can set "validate_header = false" and "validate_key = false" to disable verification of those specific parts: https://github.com/desultory/ugrd/blob/ ... ml#L89-L95

I'm not sure how the device names and uuids could be changing? The way ugrd reads this info is by checking "/sys/devices/virtual/block/dm-x/dm/name" for all devices. Between reboots the naming could change but once booted, the names and uuids should be stable. I've been strongly considering some way for aliases to be defined for devices so you can set a name you want, but detection will work if it's not currently defined exactly like that (lets you change the unlocked names between reboots sanely)