Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
dracut+zfs root+dmcrypt+systemd
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
KShots
Guru
Guru


Joined: 09 Oct 2003
Posts: 490
Location: Florida

PostPosted: Wed May 02, 2018 7:59 pm    Post subject: dracut+zfs root+dmcrypt+systemd Reply with quote

I'm running into some rather odd problems... I'm doing a systemd-boot image with a dracut initrd under a 4.9.95 kernel that boots... inconsistently... depending on how it was generated.

If I generate the image from systemrescuecd (I'll look up which kernel version later), I can generate a bootable image/kernel in a systemd boot efi package that works if I manually modify the initrd to remove the 'root=/dev/block/' from the kernel parameters that dracut insists on adding... though it imports the ZFS mounts to / rather than /sysroot, so I have to go to the emergency shell, export it, then re-import it to /sysroot. It then boots properly.

If I generate the image from the running kernel after successfully booting, the resulting image fails to find /dev/root (times out looking for "dev-gpt\x2dauto\2droot.device") and cannot be recovered without booting from the systemrescuecd kernel/userspace, after which an image may be generated like above.

I have two machines with this issue, one booting with a systemd boot image, the other through xen as a dom0 host using the xen efi loader. Both are recoverable via systemrescuecd.

When I boot the systemrescuecd, I go through the below process:

Code:

// decrypt root volumes
cryptsetup luksOpen /dev/nvme0n1p6 luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
<enter password>
cryptsetup luksOpen /dev/sdb2 luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc
<enter password>
// import the ZFS pool to /sysroot
zpool import -R /sysroot -f -d /dev/mapper rpool
// mount boot device
mount /dev/nvme0n1p1 /sysroot/boot/efi
// mount proc
mount -t proc proc /sysroot/proc
// mount dev & sys
for x in dev sys ; do
mount --rbind /$x /sysroot/$x
mount --make-rslave /sysroot/$x
done
// mount /tmp
mount -t tmpfs tmpfs /sysroot/tmp
// chroot into the environment
chroot /sysroot /bin/bash

I then run a script that generates the kernel image for me that I've standardized across the machines on my net. Effectively, I:

1. build the kernel
2. build and install kernel modules
3. generate a dracut initrd
4. generate a systemd-boot efi package
5. sign the package for secure-boot

1 & 2 above I'll skip as I doubt the issue is there. For dracut (3), I have the following config:
Code:
add_dracutmodules+="crypt systemd zfs"
hostonly="yes"
add_drivers+="nvme"
omit_drivers+="nvidia"
omit_dracutmodules+="plymouth"

I generate a dracut initrd with the following command-line parameters:
Code:
dracut -M -v --force --ro-mnt --no-compress --kernel-cmdline "init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs" /boot/initramfs-4.9.95-gentoo 4.9.95-gentoo

When booting from the 4.9.95 kernel, I get the following output (clipping out the installed files to make post shorter):
Code:
dracut: Executing: /usr/bin/dracut -M -v --force --ro-mnt --no-compress --kernel-cmdline "init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs" /boot/initramfs-4.9.95-gentoo 4.9.95-gentoo
dracut: dracut module 'bootchart' will not be installed, because command '/sbin/bootchartd' could not be found!
dracut: dracut module 'dash' will not be installed, because command '/bin/dash' could not be found!
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc (which resolves to /dev/dm-0)
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc (which resolves to /dev/dm-1)
dracut: zfsexpandknowledge: block devices backing ZFS dataset /: /dev/dm-0
/dev/dm-1
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/nvme0n1p6
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/sdb2
dracut: zfsexpandknowledge: host device /dev/nvme0n1p1
dracut: zfsexpandknowledge: host device /dev/dm-0
dracut: zfsexpandknowledge: host device /dev/nvme0n1p6
dracut: zfsexpandknowledge: host device /dev/dm-1
dracut: zfsexpandknowledge: host device /dev/sdb2
dracut: zfsexpandknowledge: device /dev/dm-1 of type zfs_member
dracut: zfsexpandknowledge: device /dev/dm-0 of type zfs_member
dracut: zfsexpandknowledge: device /dev/sdb2 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p6 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p1 of type vfat
dracut: dracut module 'plymouth' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsistart' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'dash' will not be installed, because command '/bin/dash' could not be found!
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc (which resolves to /dev/dm-0)
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc (which resolves to /dev/dm-1)
dracut: zfsexpandknowledge: block devices backing ZFS dataset /: /dev/dm-0
/dev/dm-1
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/nvme0n1p6
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/sdb2
dracut: zfsexpandknowledge: host device /dev/nvme0n1p1
dracut: zfsexpandknowledge: host device /dev/dm-0
dracut: zfsexpandknowledge: host device /dev/nvme0n1p6
dracut: zfsexpandknowledge: host device /dev/dm-1
dracut: zfsexpandknowledge: host device /dev/sdb2
dracut: zfsexpandknowledge: device /dev/dm-1 of type zfs_member
dracut: zfsexpandknowledge: device /dev/dm-0 of type zfs_member
dracut: zfsexpandknowledge: device /dev/sdb2 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p6 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p1 of type vfat
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsistart' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
bash
systemd
systemd-initrd
i18n
dracut: i18n_vars not set!  Please set up i18n_vars in  configuration file.
dracut: No KEYMAP configured.
crypt
dm
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
kernel-modules
zfs
rootfs-block
terminfo
udev-rules
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut-systemd
usrmount
base
fs-lib
shutdown
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing GenuineIntel.bin ****
dracut: *** Store current command line parameters ***
dracut: Stored kernel commandline:
dracut:  init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs
dracut:  rd.luks.uuid=luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc rd.luks.uuid=luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
dracut:  root=/dev/block/ rootfstype=zfs rootflags=rw,noatime,xattr,posixacl
dracut: ro
dracut: *** Creating image file '/boot/initramfs-4.9.95-gentoo' ***
dracut: Image: /var/tmp/dracut.M0l8kY/initramfs.img: 512
As you can see, this generates a bogus kernel commandline parameter for the root= parameter (root=/dev/block/), which doesn't exist... so because dracut doesn't allow you to override this, I have to go in and modify it manually by decompressing the initrd, changing the commandline dracut passes to the kernel, and regenerating the initrd:
Code:
// extract the initrd
cat "/boot/initramfs-4.9.95-gentoo" | while cpio -i ; do : ; done
// modify the kernel parameter passed by dracut
sed -i "s|root=/dev/block/ ||" etc/cmdline.d/95root-dev.conf
// regenerate the initrd
find . | cpio --create --format='newc' 2>/dev/null | xz --check=crc32 > "/boot/initramfs-4.9.95-gentoo"
I've done this trick for over a year now successfully, although I appear to be the only one on the internet whose dracut insists on mounting /dev/block/ as a root device...

Anyways, I then generate the systemd boot image:
Code:
mkdir /tmp/boot
cd /tmp/boot
cp /root/cmdline.txt .
ln -s /usr/src/linux/arch/x86/boot/bzImage vmlinuz
ln -s /boot/initramfs-4.9.95-gentoo initrd
objcopy \
        --add-section .osrel="/etc/osrelease" --change-section-vma .osrel=0x20000 \
        --add-section .cmdline="cmdline.txt" --change-section-vma .cmdline=0x30000 \
        --add-section .linux="vmlinuz" --change-section-vma .linux=0x40000 \
        --add-section .initrd="initrd" --change-section-vma .initrd=0x3000000 \
        /usr/lib64/systemd/boot/efi/linuxx64.efi.stub kernel.efi

I then sign this kernel:
Code:
sbsign --key /etc/efikeys/db.key --cert /etc/efikeys/db.crt --output /boot/efi/kernel.efi kernel.efi
The contents of cmdline.txt:
Code:
rd.luks.uuid=luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
rd.luks.uuid=luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc
modprobe.blacklist=nouveau
My zpool bootfs:
Code:
graendal ~ # zpool get bootfs
NAME   PROPERTY  VALUE              SOURCE
hpool  bootfs    -                  default
rpool  bootfs    rpool/ROOT/gentoo  local
So... I perform the exact same steps when running on systemrescuecd or on the successfully booted kernel/initrd. Why do I get different results?

For reference, I'm using the following software versions:

sys-kernel/gentoo-sources-4.9.95
sys-kernel/dracut-9999
sys-fs/zfs-kmod-9999
sys-fs/zfs-9999
sys-kernel/spl-9999
sys-apps/systemd-236-r5

I switched to the later version of dracut thinking it may solve my zfs issues, but the problems are the same whether I'm using 9999 or version 047-r1. I don't think I can safely downgrade zfs to non-head as one of my non-root zpools use the full feature set. I might be able to get rid of the dmcrypt layer to simplify the stack, as zfs now supports encryption, but I haven't found any means of getting a sufficiently recent zfs instance on a live environment to regenerate my zpools with encryption support.

This problem has been plaguing me for many months, maybe even a year. If anyone has any suggestions that could get this to work in a more automated fashion, I'd really appreciate it...
_________________
Life without passion is death in disguise
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum