| View previous topic :: View next topic |
| Author |
Message |
KShots
Guru
Joined: 09 Oct 2003
Posts: 591
Location: Florida
|
 Posted: Wed May 02, 2018 7:59 pm Post subject: dracut+zfs root+dmcrypt+systemd [SOLVED] |
 |
|
I'm running into some rather odd problems... I'm doing a systemd-boot image with a dracut initrd under a 4.9.95 kernel that boots... inconsistently... depending on how it was generated.
If I generate the image from systemrescuecd (I'll look up which kernel version later), I can generate a bootable image/kernel in a systemd boot efi package that works if I manually modify the initrd to remove the 'root=/dev/block/' from the kernel parameters that dracut insists on adding... though it imports the ZFS mounts to / rather than /sysroot, so I have to go to the emergency shell, export it, then re-import it to /sysroot. It then boots properly.
If I generate the image from the running kernel after successfully booting, the resulting image fails to find /dev/root (times out looking for "dev-gpt\x2dauto\2droot.device") and cannot be recovered without booting from the systemrescuecd kernel/userspace, after which an image may be generated like above.
I have two machines with this issue, one booting with a systemd boot image, the other through xen as a dom0 host using the xen efi loader. Both are recoverable via systemrescuecd.
When I boot the systemrescuecd, I go through the below process:
| Code: |
// decrypt root volumes
cryptsetup luksOpen /dev/nvme0n1p6 luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
<enter password>
cryptsetup luksOpen /dev/sdb2 luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc
<enter password>
// import the ZFS pool to /sysroot
zpool import -R /sysroot -f -d /dev/mapper rpool
// mount boot device
mount /dev/nvme0n1p1 /sysroot/boot/efi
// mount proc
mount -t proc proc /sysroot/proc
// mount dev & sys
for x in dev sys ; do
mount --rbind /$x /sysroot/$x
mount --make-rslave /sysroot/$x
done
// mount /tmp
mount -t tmpfs tmpfs /sysroot/tmp
// chroot into the environment
chroot /sysroot /bin/bash
|
I then run a script that generates the kernel image for me that I've standardized across the machines on my net. Effectively, I:
1. build the kernel
2. build and install kernel modules
3. generate a dracut initrd
4. generate a systemd-boot efi package
5. sign the package for secure-boot
1 & 2 above I'll skip as I doubt the issue is there. For dracut (3), I have the following config: | Code: | add_dracutmodules+="crypt systemd zfs"
hostonly="yes"
add_drivers+="nvme"
omit_drivers+="nvidia"
omit_dracutmodules+="plymouth" |
I generate a dracut initrd with the following command-line parameters: | Code: | | dracut -M -v --force --ro-mnt --no-compress --kernel-cmdline "init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs" /boot/initramfs-4.9.95-gentoo 4.9.95-gentoo |
When booting from the 4.9.95 kernel, I get the following output (clipping out the installed files to make post shorter): | Code: | dracut: Executing: /usr/bin/dracut -M -v --force --ro-mnt --no-compress --kernel-cmdline "init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs" /boot/initramfs-4.9.95-gentoo 4.9.95-gentoo
dracut: dracut module 'bootchart' will not be installed, because command '/sbin/bootchartd' could not be found!
dracut: dracut module 'dash' will not be installed, because command '/bin/dash' could not be found!
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc (which resolves to /dev/dm-0)
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc (which resolves to /dev/dm-1)
dracut: zfsexpandknowledge: block devices backing ZFS dataset /: /dev/dm-0
/dev/dm-1
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/nvme0n1p6
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/sdb2
dracut: zfsexpandknowledge: host device /dev/nvme0n1p1
dracut: zfsexpandknowledge: host device /dev/dm-0
dracut: zfsexpandknowledge: host device /dev/nvme0n1p6
dracut: zfsexpandknowledge: host device /dev/dm-1
dracut: zfsexpandknowledge: host device /dev/sdb2
dracut: zfsexpandknowledge: device /dev/dm-1 of type zfs_member
dracut: zfsexpandknowledge: device /dev/dm-0 of type zfs_member
dracut: zfsexpandknowledge: device /dev/sdb2 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p6 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p1 of type vfat
dracut: dracut module 'plymouth' will not be installed, because it's in the list to be omitted!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsistart' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'dash' will not be installed, because command '/bin/dash' could not be found!
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc (which resolves to /dev/dm-0)
dracut: zfsexpandknowledge: pool rpool has device /dev/mapper/luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc (which resolves to /dev/dm-1)
dracut: zfsexpandknowledge: block devices backing ZFS dataset /: /dev/dm-0
/dev/dm-1
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/nvme0n1p6
dracut: zfsexpandknowledge: slave block device backing ZFS dataset /: /dev/sdb2
dracut: zfsexpandknowledge: host device /dev/nvme0n1p1
dracut: zfsexpandknowledge: host device /dev/dm-0
dracut: zfsexpandknowledge: host device /dev/nvme0n1p6
dracut: zfsexpandknowledge: host device /dev/dm-1
dracut: zfsexpandknowledge: host device /dev/sdb2
dracut: zfsexpandknowledge: device /dev/dm-1 of type zfs_member
dracut: zfsexpandknowledge: device /dev/dm-0 of type zfs_member
dracut: zfsexpandknowledge: device /dev/sdb2 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p6 of type crypto_LUKS
dracut: zfsexpandknowledge: device /dev/nvme0n1p1 of type vfat
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsistart' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
bash
systemd
systemd-initrd
i18n
dracut: i18n_vars not set! Please set up i18n_vars in configuration file.
dracut: No KEYMAP configured.
crypt
dm
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
kernel-modules
zfs
rootfs-block
terminfo
udev-rules
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut-systemd
usrmount
base
fs-lib
shutdown
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing GenuineIntel.bin ****
dracut: *** Store current command line parameters ***
dracut: Stored kernel commandline:
dracut: init=/lib64/systemd/systemd root=zfs:AUTO rootfstype=zfs
dracut: rd.luks.uuid=luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc rd.luks.uuid=luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
dracut: root=/dev/block/ rootfstype=zfs rootflags=rw,noatime,xattr,posixacl
dracut: ro
dracut: *** Creating image file '/boot/initramfs-4.9.95-gentoo' ***
dracut: Image: /var/tmp/dracut.M0l8kY/initramfs.img: 512 |
As you can see, this generates a bogus kernel commandline parameter for the root= parameter (root=/dev/block/), which doesn't exist... so because dracut doesn't allow you to override this, I have to go in and modify it manually by decompressing the initrd, changing the commandline dracut passes to the kernel, and regenerating the initrd: | Code: | // extract the initrd
cat "/boot/initramfs-4.9.95-gentoo" | while cpio -i ; do : ; done
// modify the kernel parameter passed by dracut
sed -i "s|root=/dev/block/ ||" etc/cmdline.d/95root-dev.conf
// regenerate the initrd
find . | cpio --create --format='newc' 2>/dev/null | xz --check=crc32 > "/boot/initramfs-4.9.95-gentoo" |
I've done this trick for over a year now successfully, although I appear to be the only one on the internet whose dracut insists on mounting /dev/block/ as a root device...
Anyways, I then generate the systemd boot image: | Code: | mkdir /tmp/boot
cd /tmp/boot
cp /root/cmdline.txt .
ln -s /usr/src/linux/arch/x86/boot/bzImage vmlinuz
ln -s /boot/initramfs-4.9.95-gentoo initrd
objcopy \
--add-section .osrel="/etc/osrelease" --change-section-vma .osrel=0x20000 \
--add-section .cmdline="cmdline.txt" --change-section-vma .cmdline=0x30000 \
--add-section .linux="vmlinuz" --change-section-vma .linux=0x40000 \
--add-section .initrd="initrd" --change-section-vma .initrd=0x3000000 \
/usr/lib64/systemd/boot/efi/linuxx64.efi.stub kernel.efi |
I then sign this kernel: | Code: | | sbsign --key /etc/efikeys/db.key --cert /etc/efikeys/db.crt --output /boot/efi/kernel.efi kernel.efi |
The contents of cmdline.txt: | Code: | rd.luks.uuid=luks-ed2e40ef-3644-4205-aa90-60c3a8faddfc
rd.luks.uuid=luks-901eb7f9-98b1-4ea0-aa12-75a3cba8c6dc
modprobe.blacklist=nouveau |
My zpool bootfs: | Code: | graendal ~ # zpool get bootfs
NAME PROPERTY VALUE SOURCE
hpool bootfs - default
rpool bootfs rpool/ROOT/gentoo local |
So... I perform the exact same steps when running on systemrescuecd or on the successfully booted kernel/initrd. Why do I get different results?
For reference, I'm using the following software versions:
sys-kernel/gentoo-sources-4.9.95
sys-kernel/dracut-9999
sys-fs/zfs-kmod-9999
sys-fs/zfs-9999
sys-kernel/spl-9999
sys-apps/systemd-236-r5
I switched to the later version of dracut thinking it may solve my zfs issues, but the problems are the same whether I'm using 9999 or version 047-r1. I don't think I can safely downgrade zfs to non-head as one of my non-root zpools use the full feature set. I might be able to get rid of the dmcrypt layer to simplify the stack, as zfs now supports encryption, but I haven't found any means of getting a sufficiently recent zfs instance on a live environment to regenerate my zpools with encryption support.
This problem has been plaguing me for many months, maybe even a year. If anyone has any suggestions that could get this to work in a more automated fashion, I'd really appreciate it...
_________________
Life without passion is death in disguise
Last edited by KShots on Tue Jul 24, 2018 5:49 pm; edited 1 time in total |
|
| Back to top |
|
 |
KShots
Guru
Joined: 09 Oct 2003
Posts: 591
Location: Florida
|
| Posted: Tue Jul 24, 2018 5:49 pm Post subject: |
|
|
Update:
I was able to resolve this by moving a "root=zfs:AUTO" into cmdline.txt and out of the dracut kernel parameters. Apparently, the ZFS scripts do not process dracut kernel parameters, but they do process kernel parameters...
EDIT: I was also able to remove my ZFS repackaging hack... apparently the kernel is bootable even with a root=/dev/block/ if root=zfs:AUTO is available...
_________________
Life without passion is death in disguise |
|
| Back to top |
|
|
jjackowski
n00b
Joined: 26 Apr 2019
Posts: 1
|
| Posted: Fri Apr 26, 2019 2:25 am Post subject: |
|
|
I was having the same problem recently. Your post saved me a lot of time and frustration. I still haven't seen "root=zfs:AUTO" work, but it boots now.
Thanks!
Jeff |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Fri Apr 26, 2019 10:36 am Post subject: |
|
|
sorry i didn't see this thread or else i would have responded sooner...
root=ZFS=auto should work, as long as bootfs is set on your boot pool and unset on any additional pools. that said, manually specifying the root dataset is best, no ambiguity and you don't even need to set bootfs. 
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
