Fair warning for those playing with cryptsetup/LUKS

[BradN]

[edit] Turns out it was probably my fault. Still, my point stands - when working with partition level stuff, make sure you've got backups because all it takes is one screw up and you might lose more data than you expect.


Be very careful using an initrd initial mounting approach for the root filesystem and a second encrypted partition anywhere. After creating the second encrypted partition, something got confused and directed all access to the second encrypted device to the FIRST encrypted device, completely fucking everything up.

Make a backup before trying. You've been warned.

[Hu]

I have used these successfully in the past, so this sounds like it could be a regression. Could you describe in more detail the commands you executed that resulted in data corruption? What versions of the kernel and relevant tools were you using?

[BradN]

in use at the time Kernel 2.6.39-gentoo (r0)
(LUKS partition was probably created with 2.6.37-r1)
cryptsetup-1.1.3-r3


Here is the booting arrangement:

FreeDOS hosted grub4dos on a standard PC style MBR/partition table, kernel and initrd (squashfs format) in high memory loaded ram-disk

initrd uses squashfs/xz
kernel is built nearly fully modular, with USB drivers and USB disk support built-in
filesystems reiserfs, squashfs built-in
devtmpfs is enabled and used on the initrd

file arrangement
/bin/cryptsetup
/bin/busybox
/bin/sh -> busybox
/dev
/modules
/modules/0lib-ata.ko (not 100% sure on these two filenames, building a new kernel with squashfs to check)
/modules/1ata-piix.ko
/proc
/realroot
/sbin
/sbin/init
/sys

actual contents of /sbin/init once I get a squashfs kernel set up again

Basically /sbin/init does these things:

uses devtmpfs
waits for encrypted LUKS volume to appear as a device by calling findfs until it shows up (searching by UUID)
does luksOpen with that device and name 'cryptstick'
mounts /dev/mapper/cryptstick to /realroot

executes a fun procedure to switch /realroot to be the new root filesystem, with /realroot/tmp holding the initrd, then unmounting the initrd using sh from the real root partition


Basically the problem occurred as I was trying to use my encrypted USB stick to permanently install the environment to the laptop's HDD it was running from. I had luksFormat'd the new partition and then opened it (I'm 99% sure I had the right device identifier, but if I hadn't, should it have warned me that the device was already in use?), and then at that point mkreiserfs on /dev/mapper/cryptdisk (the second device set up for the main HDD's encrypted root) would actually be writing to /dev/mapper/cryptstick. I could see the light on the memory stick blink and everything.

Unfortunately the machine's not running anymore and the filesystem lost so I can't get the bash history.

I'm nearly positive I arrowed up and double checked the devices specified afterwards (/dev/sda was the memory stick, and /dev/sdb was the internal hdd).

Actually... now that I think about that aspect... it's possible I had booted the system initially from the hard drive and not the memory stick at that point - the devices could have been reversed due to the newer initrd that instantly loaded the ata-piix driver before the USB drivers had registered the USB stick.

So maybe I had double checked that the device was what I was expecting, just I was expecting the wrong device. That's my best theory right now. But should cryptsetup allow mounting the same volume twice at the same time?

[BradN]

Update: Here's the init script:

[Hu]

[BradN]

Using "exec" in the error path: I had thought about this, but I figured if something went wrong, I would be preserving the most usefulness of the environment by starting a fully fresh shell - if exec'ing the new /sbin/init is to be possible, it can only be done by the first process. I didn't want that first process to be stuck in the middle of executing a script.

[balkira]

Hi,

1) why making an initrd (deprecated) and not an initramfs, the later being more robust and easier to maintain?
2) pivot_root usually is for initrd, switch_root for initramfs. You should use switch_root.
3) you should use the Gentoo's linuxrc scripts as a base by default and then expand (or adapt) its features (if needed) from there.
4) switch_root+squashfs is ok.
5) to use xz with an initramfsf you need the kernel builtin for it.

I totally understand the need of a custom initramfs but why not using the linuxrc from Gentoo (the one shipped in genkernel)? It handles most of every need for a custom initramfs (passing kernel options for extra features) AND you still have a somewhat 'official' boot kernel interface (at least you can apply the Gentoo docs for the mystical boot options). Drop it inside your initramfs it just works (don't forget initrd.scripts and initrd.defaults to put in /etc for the linuxrc to work).
I'm using cryptsetup for a few years already and never experienced data corruption (else than corruption of the disk itself).

My laptop has 2 SSD disks. The first one is / the second is /home. I use cryptsetup-1.2.0-r1.
When booting the initramfs opens / and after switch_root /home gets opened (so I type 2 passphrases).

I've rewritten 90% of genkernel's features (and a few genkernel doesn't have - like adding dropbear to the initramfs to reboot fully luks'd remote servers via ssh) in a python project called KIGen and since then I've never had a single headache about cryptsetup and/or initramfs (I use it weekly for the past year almost).
You could try using https://github.com/r1k0/kigen (latest 0.3.0 ebuild in download section) it's user friendly and has a lot of configurable features - lvm2,disklabel,dmraid,dropbear for remote SSH booting,keymap,splash and others - and creates a rich / environment from which you can boot painlessly.
Note that the whole initramfs fs is readable in /var/tmp/kigen/work after each build so you can create a consistent base and then finish the work manually if you want or let the software handle the job for you.

Good luck
_________________
no way to happiness, happiness is the way

[BradN]

I'm pretty sure the data loss was my fault at this point, especially since retesting and finding cryptsetup does allow format/open on a device already in use.

I'm a little confused what the difference between initrd / initramfs actually is? I've seen references to making a cpio archive, and it seems like this route would actually work with switch_root. What exactly is an initramfs?


To point out what I was talking about with switch_root, here is some of the kind of code busybox has for its switch_root:

[Hu]

Since squashfs is read-only, I do not think you can clear it out. For kernels where I want an initramfs, I let the kernel build it into the kernel image. I do not compress the initramfs explicitly, but do benefit from it being in the kernel image, which is then compressed.

[BradN]

I was mostly replying to balkira's comment:

[Hu]

Yes. You can also specify a file that describes a virtual folder, which will be constructed and embedded. This can be useful if you do not want to keep separate copies of all the files that you include.