Lvm-Partitions vanished with Luks setup

[Manu311]

Hello,

I have a pretty bad problem with my Luks/Lvm setup and lost access to my system. Hopefully someone can help me get back my data.

I'll start with my setup:
I needed to have my Gentoo on an (almost) fully encrypted harddrive with as much security as possible, so I went with luks (aes-xts-plain64) with detached headers on an usb-stick. Inside of that luks there's a lvm with two partitions - one ext4 and one swap partition.
Since genkernel couldn't handle that, I switched to dracut (which I had to slighly manipulate so it waits for the usb-device).
I'm aware that the kernel and initramfs aren't encrypted so they are vulnerable, however that's not my problem right now (actually I could live with a bit less security).

So what happened:
The system has been working like this for half a year and I failed to setup any backups (I guess most important data is on servers anyways, so the loss is only 1-2 weeks of work instead of months). Since I had to replace my keyboard, I had to shutdown the notebook, which I did. Since I haven't changed any keyboards on notebooks, I had a couple of trial-and-errors where I cut the power during boot or kernel bootup because the keyboard didn't work (and I didn't have the patience to wait for the password prompt).
Eventually the keyboard worked and I wanted to boot back up - however after entering the password, dracut stopped and dumped me into it's backup shell. I don't recall exactly what I tried there but without much problems I could decrypt the partition again however LVM didn't find anything within it.
I recalled having such problem in the past (like 5 month ago) and that the solution was simply to recreate the Pv with the exact same uuid and restore my config (which I had backuped months ago) - anyways I searched for a tutorial and found this: https://www.golinuxcloud.com/recover-lv ... a_in_Linux
which I followed.
At least step 1-4 were no problem, however step 5 didn't work. Neither did anything I have tried after that.

Things I tried:
0. Step 1-4 in the above mentioned tutorial
1. simply mounting: "wrong fs type, bad option, bad superblock on /dev/mapper/......., missing codepage or helper program, or other errors"
2. testdisk deep analysis without partition table: nothing found
3. testdisk deep analysis with intel partition table: found hundreds of partitions it can't recover, most of them bigger than the actual disk (just garbage data)

[alamahant]

Try from a live cd.
Are the lvm partitions visible from livecd?
Please run lsblk
If the partitions are visible maybe your problem is just reinstalling grub.
Can you mount the root partition from a livecd?
Lvm partitions dont just disappear...

[Manu311]

Try from a live cd.
Are the lvm partitions visible from livecd?
Please run lsblk
If the partitions are visible maybe your problem is just reinstalling grub.
Can you mount the root partition from a livecd?
Lvm partitions dont just disappear...

After a couple of minutes, I switched to a "livecd" (it's just the minimal installation usb-stick I created a while back) with gentoo on it. Everything's the same there.
lsblk and lvdisplay look normal since I've restored the backup lvm-configuration file. I guess it would have recreated the partitions there even if I would have overwritten all their bytes.

Code: Select all

  --- Logical volume ---
  LV Path                /dev/censored-encrypted/swap_fs
  LV Name                swap_fs
  VG Name                censored-encrypted
  LV UUID                I-have-no-idea-if-this-is-sensible-data
  LV Write Access        read/write
  LV Creation host, time livecd, 2020-10-14 14:36:59 +0000
  LV Status              available
  # open                 0
  LV Size                <37.36 GiB
  Current LE             9564
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/censored-encrypted/root_fs
  LV Name                root_fs
  VG Name                censored-encrypted
  LV UUID                I-have-no-idea-if-this-is-sensible-data-either
  LV Write Access        read/write
  LV Creation host, time livecd, 2020-10-14 14:37:00 +0000
  LV Status              available
  # open                 0
  LV Size                <916.01 GiB
  Current LE             234498
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
   
livecd ~ # lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                               7:0    0 533.2M  1 loop  /mnt/livecd
sda                                 8:0    1   3.7G  0 disk  
|-sda1                              8:1    1   3.6G  0 part  /mnt/cdrom
`-sda2                              8:2    1    20M  0 part  
nvme0n1                           259:0    0 953.9G  0 disk  
|-nvme0n1p1                       259:1    0   511M  0 part  /mnt/boot
`-nvme0n1p2                       259:2    0 953.4G  0 part  
  `-censored-crypt                253:0    0 953.4G  0 crypt 
    |-censored--encrypted-swap_fs 253:1    0  37.4G  0 lvm   
    `-censored--encrypted-root_fs 253:2    0   916G  0 lvm   

[alamahant]

I would advice you if your root partition can be mounted to backit up somewhere using rsync and then delete and recreate your pv vg and lvs...and move your backup to your newly created lvm partition.
And maybe if possible avoid taking such drastic actions like the article you are following...

[Manu311]

I would advice you if your root partition can be mounted to backit up somewhere using rsync and then delete and recreate your pv vg and lvs...

I would love to do that, but my root partition is the only data-partition inside of that lvm, so I can't back it up unless I can fix my problem at least partially.

I could probably backup the entire disk, but with my connection it will take days to transfer 1 TB (and I don't have another storage device lying around rn)

As usual backups would have saved my day - buuuuut it's too late now. I blame the me from the past

[alamahant]

But did you try to mount the root lvm from the livecd?
Is it mountable?

[Manu311]

But did you try to mount the root lvm from the livecd?
Is it mountable?

Like this?:

Code: Select all

livecd ~ # mount /dev/mapper/censored--encrypted-root_fs /mnt/gentoo/
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/mapper/censored--encrypted-root_fs, missing codepage or helper program, or other error.

[NeddySeagoon]

Manu311,

Try

Code: Select all

dumpe2fs /dev/mapper/censored--encrypted-root_fs

Does that look like ext4?

Then there is

Code: Select all

mount -o ro,sb=131072 /dev/mapper/censored--encrypted-root_fs /mnt/gentoo/

Then there is the offset=<Bytes> option, where Bytes is the offset to the start of the filesystem. That's difficult to discover on LVM as you seed to knom the on disk layout.
However, your /dev/mapper/censored--encrypted-root_fs is a single extent, so that can work.

There are other values of sb= you can try too.

[Manu311]

Manu311,

Try

Code: Select all

dumpe2fs /dev/mapper/censored--encrypted-root_fs

Does that look like ext4?

Then there is

Code: Select all

mount -o ro,sb=131072 /dev/mapper/censored--encrypted-root_fs /mnt/gentoo/

Then there is the offset=<Bytes> option, where Bytes is the offset to the start of the filesystem. That's difficult to discover on LVM as you seed to knom the on disk layout.
However, your /dev/mapper/censored--encrypted-root_fs is a single extent, so that can work.

There are other values of sb= you can try too.

Sadly I get this:

Code: Select all

livecd ~ # dumpe2fs /dev/mapper/censored--encrypted-root_fs 
dumpe2fs 1.45.5 (07-Jan-2020)
dumpe2fs: Bad magic number in super-block while trying to open /dev/mapper/censored--encrypted-root_fs
Couldn't find valid filesystem superblock.

Putting sb-numbers there just results in the same error all the time.

Code: Select all

mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/mapper/censored--encrypted-root_fs, missing codepage or helper program, or other error.

[NeddySeagoon]

Manu311,

Either your filesystem has been destroyed or /dev/mapper/censored--encrypted-root_fs does not point to the start of it, that would explain why the alternate superblock mount failed too.

The LVM recovery process only writes PV and LV metadata. filesystems are not harmed.
You also have censored--encrypted-swap_fs,

Does

Code: Select all

swapon /dev/mapper/censored--encrypted-swap_fs

work?
That would be encouraging as it would show the swap signature is still where you expected it to be.

[Manu311]

Manu311,

Either your filesystem has been destroyed or /dev/mapper/censored--encrypted-root_fs does not point to the start of it, that would explain why the alternate superblock mount failed too.

The LVM recovery process only writes PV and LV metadata. filesystems are not harmed.
You also have censored--encrypted-swap_fs,

Does

Code: Select all

swapon /dev/mapper/censored--encrypted-swap_fs

work?
That would be encouraging as it would show the swap signature is still where you expected it to be.

That's kinda what I am assuming as well. Somehow the headers are corrupted, but I have no idea why. If I understand it correctly, they should have been able to be restored with the backup file I had on my usb-stick.

I tried swapon, it fails with "read swap header failed".

In my head I should be able to get the data back with some tool since it shouldn't be encrypted anymore.
Unless for some reason the luks-decryption destroyed something or rather isn't decrypting correctly anymore. I might have had a typo when I manually unlocked the drive.
The command should be "cryptsetup luksOpen --header headerfile-wherever-it-may-be /dev/nvme0n1p2 censored-crypt". So I'm not sure what would happen if I had a typo somewhere in that command. I assume it would just fail to decrypt (or not accept the correct password).

[NeddySeagoon]

Manu311,

Do you know some long (the longer the better) filenames you cat search for?

Try

Code: Select all

strings /dev/mapper/censored--encrypted-root_fs | grep -i <long_file_name>

or even just

Code: Select all

strings /dev/mapper/censored--encrypted-root_fs

and see what you find.

The idea is to locate the start of the filesystem, so it can be mounted.

[Manu311]

The idea is to locate the start of the filesystem, so it can be mounted.

Interesting - the gentoo minimal image apparently doesn't have "strings" on it. And since there's no emerge either, I figured the idea should still work if I just use grep on it (but it will probably take longer), so that's running right now.

I somehow assume I'm not gonna find anything (because I'm out of luck this week), but even if I did, I have no idea how I would find the offset from the output.
Anyways, I'll report back, once the grep ran through (probably half an hour).

Another question: Should I run the grep-command on the "pv" instead? So the unencrypted partition which holds the lvm-stuff? I guess chances could be slighly better if I would search on that - but I have no idea if a text-search could actually find anything readable in either those formats.

//EDIT:
It was faster than I expected. Grep didn't print out anything, so nothing has been found

[NeddySeagoon]

Manu311,

The serching has to happen on the unlocked encrypted space, so the LUKS layer is used.
Searching the raw underlying device will do just that and is the crypto is any good at all, you won't find anything.

You are going to want more tools than there is on the Gentoo minimal CD.
Can you try the adminCD or some other random live distro?

If all that fails, all that left is thrashing through the partition layout, the PV header and the LVM metadata on disk locations to work out where your filesystem should start
Those things are unknown but are
a) constants
b) documented.

What does

Code: Select all

pvdisplay

tell?

We know that the two LVs sikes and that they a single logical extents, so that a plain mount can work, without the lvm layer to put the disjoint bits together.

Code: Select all

lvdisplay -m

may be useful.

Code: Select all

pvs --segments -o+lv_name,seg_start_pe,segtype

Your

Code: Select all

LV UUID                I-have-no-idea-if-this-is-sensible-data-either

will stop things working.
If you want to redact information tell us. It way be important. The old UUIDs and the nem UUIDs munt be identical.

[Hu]

You probably should have come to us for help before trying to restore the PV/LV metadata. As I understand your story, at that point, there were no unusual modifications to the data.

UUIDs are Universally Unique IDentifiers. You might consider them secret if you expect that someday someone will see the real UUIDs on your drive, and you want to be able to deny to them that you are Manu311 on the Gentoo forums. Other than that, sharing the UUIDs is probably fine. If you really want to keep them secret, write up a mapping table and apply that when you post. We don't necessarily need the real UUIDs, but as Neddy says, it would be helpful if we can distinguish when two posted UUIDs are the same on your disk.

What is the output of file -s device on each device that you expect to hold a filesystem/swap? If there are recognizable patterns, file can report them, even if the underlying structures are a bit damaged.

Since you are using LUKS, a misspelled command while unlocking will usually just fail to unlock, rather than reporting success and returning garbage.

[Manu311]

NeddySeagoon,

my understand is that the --encrypted_root-fs is just the partition within LVM. While the PV the lvm is on (/dev/mapper/censored-crypt) should also be decrypted. The crypted device should be /dev/nvme0n1p2

I'll download the admin-cd, so I got more tools.

I guess there's also no use in redacting uuids when I can't access my data anymore, so here is the output from the commands you asked for:

Code: Select all

livecd ~ # pvdisplay
  --- Physical volume ---
  PV Name               /dev/mapper/censored-crypt
  VG Name               censored-encrypted
  PV Size               <953.37 GiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              244062
  Free PE               0
  Allocated PE          244062
  PV UUID               JgC3cr-Ie7Q-J9T1-1huo-QH69-KW8v-I3cZrq
   
livecd ~ # lvdisplay -m
  --- Logical volume ---
  LV Path                /dev/censored-encrypted/swap_fs
  LV Name                swap_fs
  VG Name                censored-encrypted
  LV UUID                gdEwbz-2PLb-tyPY-ZzEg-Ijmq-2gET-9YBbCs
  LV Write Access        read/write
  LV Creation host, time livecd, 2020-10-14 14:36:59 +0000
  LV Status              available
  # open                 0
  LV Size                <37.36 GiB
  Current LE             9564
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Segments ---
  Logical extents 0 to 9563:
    Type		linear
    Physical volume	/dev/mapper/censored-crypt
    Physical extents	0 to 9563
   
   
  --- Logical volume ---
  LV Path                /dev/censored-encrypted/root_fs
  LV Name                root_fs
  VG Name                censored-encrypted
  LV UUID                tYLAcL-i1v1-7UpV-HHV4-d9sN-Jqc6-C7j883
  LV Write Access        read/write
  LV Creation host, time livecd, 2020-10-14 14:37:00 +0000
  LV Status              available
  # open                 0
  LV Size                <916.01 GiB
  Current LE             234498
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
   
  --- Segments ---
  Logical extents 0 to 234497:
    Type		linear
    Physical volume	/dev/mapper/censored-crypt
    Physical extents	9564 to 244061
   
   
livecd ~ # pvs --segments -o+lv_name,seg_start_pe,segtype
  PV                         VG                 Fmt  Attr PSize    PFree Start SSize  LV      Start Type  
  /dev/mapper/censored-crypt censored-encrypted lvm2 a--  <953.37g    0      0   9564 swap_fs     0 linear
  /dev/mapper/censored-crypt censored-encrypted lvm2 a--  <953.37g    0   9564 234498 root_fs     0 linear

And here's the backup file:

Code: Select all

livecd ~ # cat /mnt/cdrom/censored-encrypted-vg 
# Generated by LVM2 version 2.02.187(2) (2020-03-24): Wed Oct 14 14:37:03 2020

contents = "Text Format Volume Group"
version = 1

description = "vgcfgbackup -f /mnt/cdrom/censored-encrypted-vg"

creation_host = "livecd"	# Linux livecd 5.4.60-gentoo-x86_64 #1 SMP Mon Sep 21 04:11:09 UTC 2020 x86_64
creation_time = 1602686223	# Wed Oct 14 14:37:03 2020

censored-encrypted {
	id = "qHypt4-SPr6-szq9-XzYa-IMx6-0czK-dxUN9I"
	seqno = 3
	format = "lvm2"			# informational
	status = ["RESIZEABLE", "READ", "WRITE"]
	flags = []
	extent_size = 8192		# 4 Megabytes
	max_lv = 0
	max_pv = 0
	metadata_copies = 0

	physical_volumes {

		pv0 {
			id = "JgC3cr-Ie7Q-J9T1-1huo-QH69-KW8v-I3cZrq"
			device = "/dev/mapper/censored-crypt"	# Hint only

			status = ["ALLOCATABLE"]
			flags = []
			dev_size = 1999360000	# 953.369 Gigabytes
			pe_start = 2048
			pe_count = 244062	# 953.367 Gigabytes
		}
	}

	logical_volumes {

		swap_fs {
			id = "gdEwbz-2PLb-tyPY-ZzEg-Ijmq-2gET-9YBbCs"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			creation_time = 1602686219	# 2020-10-14 14:36:59 +0000
			creation_host = "livecd"
			segment_count = 1

			segment1 {
				start_extent = 0
				extent_count = 9564	# 37.3594 Gigabytes

				type = "striped"
				stripe_count = 1	# linear

				stripes = [
					"pv0", 0
				]
			}
		}

		root_fs {
			id = "tYLAcL-i1v1-7UpV-HHV4-d9sN-Jqc6-C7j883"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			creation_time = 1602686220	# 2020-10-14 14:37:00 +0000
			creation_host = "livecd"
			segment_count = 1

			segment1 {
				start_extent = 0
				extent_count = 234498	# 916.008 Gigabytes

				type = "striped"
				stripe_count = 1	# linear

				stripes = [
					"pv0", 9564
				]
			}
		}
	}

}
livecd ~ # 

[NeddySeagoon]

Manu311,

As you explained it, the LUKS container is on /dev/nvme0n1p2
Once you unlock the container, you have a lvm2 PV with two logical volumes inside the physical volume.
You can do it the other way round too but you have several LUKS containers then.

If thats correct, the on disk layout is your primary GPT copy, /dev/nvme0n1p1, /devnvme0n1p2 then the backup copy of the GPT.
There may be some wasted space too.

/dev/nvme0n1p2 contains the LUKS header, which you say is detached. That does not mean that the space is reallocated, the PV metadata, then LV metadata and your user data.
That's followed by the backup copy of the GPT. There may be same wasted space around there too.

From pvdisplay We know that your PV /dev/mapper/censored-crypt contains 244062, 4.00 MiB extents with 2.00 MiB at the end and its all allocated.
We also know from lvdisplay -m that /dev/censored-encrypted/root_fs is at the end of the PV.
Physical extents 9564 to 244061
In summary, /dev/censored-encrypted/root_fs starts at 234498.5 * 4MiB + Backup GPT + waste space from the end of /dev/nvme0n1.
That's easier than working forward.
We are missing two pieces of information to be able to calculate sb= for the mount command.
The first is your partition table in sectors.
fdisk or parted will show that. We need the output in sectors.
Once we have that, we can work out the offset= value.

This Wikipedia suggests that there is no wasted space. Thus the backup GPT is 2MiB and offset= is 234499 * 4MiB from the end of the volume. The 2MiB GPT and 2MiB unusable space in the PV make up another extent.

Once we have a value for offset=, we can try mounting the filesystem with respect to /dev/nvme0n1. The LUKS container will need to be unlocked but we won't use any other metadata, no partation table, no LVM, just the start of the filesystem from the start of the block device.

-- edit --

The concern about redacted data is the not knowing its redacted, in case you redact something we need to know and miss it.

[Manu311]

So, this?

Code: Select all

Disk /dev/mapper/censored-crypt: 953.38 GiB, 1023672320000 bytes, 1999360000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/censored--encrypted-swap_fs: 37.37 GiB, 40114323456 bytes, 78348288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/censored--encrypted-root_fs: 916.1 GiB, 983555899392 bytes, 1921007616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[NeddySeagoon]

Manu311,

The entire /dev/nvme0n1.

We know the filesystem start from the end of the drive but offset= is a count of bytes from the start of the drive.

[Manu311]

Manu311,

The entire /dev/nvme0n1.

We know the filesystem start from the end of the drive but offset= is a count of bytes from the start of the drive.

Alright, so here's the fdisk-part from nvme0n1:

Code: Select all

Disk /dev/nvme0n1: 953.89 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Samsung SSD 970 PRO 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device           Start        End    Sectors   Size Type
/dev/nvme0n1p1    2048    1048575    1046528   511M EFI System
/dev/nvme0n1p2 1048576 2000408575 1999360000 953.4G Linux filesystem

[DespLock]

testdisk might be of help here.

If you have internet access from your gentoo "live usb" you could simply emerge it. Otherwise SystemRescueCD is a good candidate for another live usb.

You need to open your /dev/nvme0n1p2 with cryptsetup and use testdisk on the /dev/mapper/censored-crypt

read the fine manual before trying to recover. Here is a tutorial:
https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step

[NeddySeagoon]

Manu311,

Its even easier than I though.

The drive is 1024209543168 bytes.
234499 * 4MiB is 983560093696 bytes

The difference is 40,649,449,472 which is exactly divisible by 4096,

Unlock your encrypted volume.
Then

Code: Select all

mount -o ro,offset=40649449472 /dev/nvme0n1 /mnt/<someplace>

That attempts a read only mount of the filesystem that starts 40649449472 bytes from the start of /dev/nvme0n1
The divisible by 4096 is a sanity check, so the filesystem starts on a physical sector boundary, as it should.

If that works, plan on copying your data out and remaking the PV and its content.

[Manu311]

NeddySeagoon,

you gave me hope. But then my system destroyed it again:

Code: Select all

livecd ~ # mount -o ro,offset=40649449472 /dev/nvme0n1 /mnt/gentoo/
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.

But then also nvme0n1 is not the decrypted device, it's the decrypted, right? The decrypted should be censored-crypt.


DespLock,

I have tried testdisk already, but it doesn't find anything.

[NeddySeagoon]

Manu311,

nvme0n1 is the the entire block device. I was expecting the kernel to use the decryption layer once the LUKS volume was opened.
It looks like I was mistaken as

Code: Select all

mount -o ro,offset=40649449472 /dev/nvme0n1 /mnt/gentoo/ 

ignores the crypto.
Its interesting that it returns an error about bad superblock on /dev/loop1. I don't understand why loop1 is there.

Lets try a different offset=, from the start of /dev/mapper/censored-crypt

/dev/mapper/censored-crypt is 1999360000 sectors, that's the same size as /dev/nvme0n1p2, so they must describe the same rectors on the drive as they are at the end of the drive.
/dev/nvme0n1p2 starts at 1048576 thus /dev/mapper/censored-crypt must start there too.

We had offset=40649449472 but we need to reduce that by 1048576*512
I get 40112578560.

That gets us to

Code: Select all

mount -o ro,offset=40112578560 /dev/mapper/censored-crypt /mnt/gentoo/ 

You can also add in the sb=131072 option to either command to try an alternate superblock.

[Manu311]

NeddySeagoon,

sadly no luck either:

Code: Select all

livecd ~ # mount -o ro,offset=40112578560 /dev/mapper/censored-crypt /mnt/gentoo/
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.
livecd ~ # mount -o ro,offset=40112578560,sb=131072 /dev/mapper/censored-crypt /mnt/gentoo/
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.