| View previous topic :: View next topic |
| Author |
Message |
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
 Posted: Wed Nov 12, 2014 3:50 am Post subject: Recover partly overwritten luks volume? |
 |
|
Recover partly overwritten luks volume?
---
What follows below is the entire problem on my hands now, or if you like, the story, in Unix terms.
I've mouse over to select what happened as the urxwt terminal showed it to me, and edited the paste to present the exact problem the best I could.
The ukra is not my user name, but, to not disturb the spacing that was there in original, I shortened ukrainian (which I like to put instead of my real username) to ukra. "ua" is two-letter code for Ukraine, country of the nation related to us Croats, su "uabox" I use.
Also, some other not-so-relevant data, such as mount points, ip addresses, not much else, is altered, but I did not touch the lsusb output and the numbers in the listing showing the drastic change that froze my life with angst for a while.
| Code: |
uabox ukra # ls -l /mnt/c1
total 565657156
...[snip]...
listing here of some other HDD unrelated to the issue
...[snip]...
uabox ukra # ls -l /mnt/b1
total 565657156
...[snip]...
listing here of some other HDD unrelated to the issue
...[snip]...
uabox ukra # umount /mnt/c1
uabox ukra # umount /mnt/b1
uabox ukra # lsusb
Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 039: ID 2537:1066
Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 007: ID 152d:2509 JMicron Technology Corp. / JMicron USA Technology Corp. JMS539 SuperSpeed SATA II 3.0G Bridge
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
uabox ukra #
|
I leave these mostly intact, just in case, but these don't matter much yet:
| Code: |
uabox ukra # mount
rootfs on / type rootfs (rw)
...[snip]...
devtmpfs on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=2040363,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nodev,relatime,size=1638992k,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
mdev-tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
shm-tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
...[snip]...
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nodev,noexec,nosuid)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw,nodev,noexec,nosuid)
192.168.9.39:/sw3 on /mnt/some-where type nfs (rw,addr=192.168.9.39)
...[snip]...
uabox ukra # mount | grep sd
/dev/sda3 on / type ext4 (rw,noatime,data=ordered)
nfsd on /proc/fs/nfsd type nfsd (rw,nodev,noexec,nosuid)
/dev/sde5 on /mnt/some-where2 type ext4 (rw)
uabox ukra #
|
Here I went to disconnect the other two HDDs unrelated to this problem, and to connect the HDDs where the human error of the worse kind (unless it shows recoverable) has happened to me.
So here I attached two HDDs, on the two different types of USB-SATA3 adaptors, JMicron one, and the other represented with "Bus 010 Device [...]: ID 2537:1066" above/below.
I left those as in the original, althouth they are probably not relevant much to the problem at all either. I have a file recovery problem. Read on. As I said, I leave these as in original:
| Code: |
uabox ukra # lsusb
Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 041: ID 2537:1066
Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 019: ID 152d:2509 JMicron Technology Corp. / JMicron USA Technology Corp. JMS539 SuperSpeed SATA II 3.0G Bridge
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
uabox ukra #
|
Below, of course, is just me pressing tab, to see what I can mount from among my devices (and I find that...
| Code: |
uabox ukra # ls -l /dev/sd
sda sda2 sda4 sdb sdc sde sde2 sde4 sde6
sda1 sda3 sda5 sdb1 sdc1 sde1 sde3 sde5
|
... sdb1 and sdc1 are new and on the two HDDs that I connected.
| Code: |
uabox ukra # ls -l /dev/sdb1 /mnt/b1/
uabox ukra # brw-rw---- 1 root disk 8, 17 2014-11-11 23:18 /dev/sdb1
/mnt/b1/:
total 0
uabox ukra #
|
And I mount them.
| Code: |
uabox ukra # mount /dev/sdb1 /mnt/b1/ &
[1] 1635
uabox ukra #
[1]+ Done mount /dev/sdb1 /mnt/b1/
uabox ukra # mount /dev/sdc1 /mnt/c1/ &
[1] 1638
uabox ukra #
[1]+ Done mount /dev/sdc1 /mnt/c1/
uabox ukra # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 66G 44G 18G 72% /
mdev-tmpfs 10M 60K 10M 1% /dev
tmpfs 1.6G 148K 1.6G 1% /run
cgroup_root 10M 0 10M 0% /sys/fs/cgroup
shm-tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/mapper/volugrp 1.7T 1.4T 150G 91% /sw3
192.168.9.39:/sw3 1.7T 580G 990G 37% /mnt/some-where
/dev/sde5 1.7T 803G 789G 51% /mnt/some-where2
/dev/sdb1 2.7T 1.2T 1.4T 47% /mnt/b1
/dev/sdc1 2.7T 1.2T 1.4T 48% /mnt/c1
uabox ukra #
|
We have reached the point where the problem will occur. Pls. take notice of the file H_E09.vol which is some 430G heavy, in both the mounted HDDs.
| Code: |
uabox ukra # ls -l /mnt/b1/
total 909314276
...[snip]...
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:26 H_E09.mnt
-rw-r--r-- 1 ukra ukra 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 465567744000 2014-09-11 23:09 H_E09.vol
...[snip]...
uabox ukra # ls -l /mnt/c1/
total 914819664
...[snip]...
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:27 H_E09.mnt
-rw-r--r-- 1 ukra ukra 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 465567744000 2014-09-11 23:07 H_E09.vol
...[snip]...
uabox ukra #
|
That file is (alas!, it was) a luks volume. Alas!, it was a luks volume, full of data:
| Code: |
-rw-r--r-- 1 ukra ukra 35810 2014-09-11 22:37 H_E09.sum
|
contains the hashes of those data, currently lost for me (some two months worth of work).
The:
| Code: |
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
|
however, is the luks header backup, taken with the command:
| Code: |
# cryptsetup luksHeaderBackup H_E09.vol --header-backup-file H_E09.bak
|
and the H_E09.bak.sum is just its sum, stowed in other places too. So, maybe all is not lost, because the header should be possible to place it back, or whatever the term for doing it.
I'm saying all this, because, and here comes the horror of human error...
Because, being emerged in other work, and pushing myself to work faster, I did the usual procedure in a rush, and didn't notice that I I already had a file of that freaking name H_E09.vol, and went on creating exactly another file-to-be-luks-volume by, the horror, that same name!
This is me searching for previous command when I used the same dd create empty file... command, I searched for 111000000:
| Code: |
(reverse-i-search)`dd': ^C if=E1108_nbg_sda2.dd of=/dev/sdd2 &
(failed reverse-i-search)`111000000': echo ^C1000000000*4096|bc
uabox ukra # ^C
uabox ukra # dd if=/dev/zero bs=4k count=27750000 of=E_E12.vol & ^C
uabox ukra # cd /mnt/c1/
uabox c1 # echo 111000000*4096|bc
454656000000
|
And that is why I searched for it. It gives the right measure for me. Actually the volume that I deleted was made with the count=113664000 (do the echo 113664000*4096|bc and it will give you exactly the size of the deleted luks volume of mine), but I decided that 111000000 was the better measure for me.
And here I go to overwrite my luks volumes full of my two months worth of data...
| Code: |
uabox c1 # dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol &
[1] 1665
uabox c1 # cd /mnt/b1/
/mnt/b1
uabox b1 # dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol &
[2] 1669
uabox ukra #
|
However, it's not just sheer recklessnes. I've been losing my eyesight and other
ailments and some occasional distress contribute a lot...
| Code: |
uabox b1 # cd ~ukra
uabox ukra # ls -l started
-rw-r--r-- 1 root root 0 2014-11-10 05:30 started
uabox ukra # touch started
|
This is just a notch, for me to easily see how long the process is taking, by comparing timestamps:
| Code: |
uabox ukra # ls -l started
-rw-r--r-- 1 root root 0 2014-11-11 23:21 started
uabox ukra #
|
Make sure now that you notice the sizes and the modification times of the new, overwriting the old, files H_E09.vol:
| Code: |
uabox ukra # ls -l /mnt/b1/
total 456289892
...[snip]...
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:26 H_E09.mnt
-rw-r--r-- 1 ukra ukra 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 1670819840 2014-11-11 23:21 H_E09.vol
...[snip]...
uabox ukra # ls -l /mnt/c1/
total 464777784
...[snip]...
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:27 H_E09.mnt
-rw-r--r-- 1 ukra ukra 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 4724903936 2014-11-11 23:21 H_E09.vol
...[snip]...
uabox ukra #
|
These sizes are not "465567744000" as previously, and sure the timestamps have changed... There I now notice what I have done! I must have gone pale...
A second or two, and I decided that...:
| Code: |
uabox ukra # kill -9 %1
uabox ukra # kill -9 %2
uabox ukra #
uabox ukra # jobs
[1]- Running dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol & (wd: /mnt/c1)
[2]+ Running dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol & (wd: /mnt/b1)
uabox ukra #
[1]- Killed dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol (wd: /mnt/c1)
(wd now: /home/ukra)
[2]+ Killed dd if=/dev/zero bs=4k count=1110000000 of=H_E09.vol (wd: /mnt/b1)
(wd now: /home/ukra)
uabox ukra #
|
...I had decided some 10 (ten) to 30 (thirty) seconds ago that killing those two processes dead was the sole option to do... It took a little to accept my command, dd takes some time (around 20 seconds in my case here) to be killed.
The next obvious two commands to do, I actually did after I went for a walk (and a prayer, because it's two months of my work hanging, and I am hopeful there must be a way to recover most of those volumes' content, but I am surely not certain that I will be able to recover those)...
So, the next two commands I did after another half hour or more.
| Code: |
uabox ukra # umount /mnt/b1 & umount /mnt/c1 &
[1] 1909
[2] 1910
uabox ukra #
[1]- Done umount /mnt/b1
uabox ukra #
[2]+ Done umount /mnt/c1
uabox ukra #
|
And no more there to be told about this easy to present issue, in effect, yes, it is not complicated to deploy the exact thing that happened.
I now don't intend to rush to solve this.
The HDDs, both of them, are still attached to the same adaptors.
I'm going to post this here, on Gentoo Forums, even though other places could be more appropriate, such as various data recovery forums, because, on another thought, this should be solveable...
Because this should be solveable... The header is backed up...
And of all the 465567744000 of mostly data (plus some overhead that luks needs) 1670819840 in one disk, and 4724903936 in the other will not be recoverable, because they are overwritten... That is some 1.6G on one disk and 4.5G on the other, of the 430G on each respectively, will not be recoverable. The rest should be, somehow, recoverable. Somehow...
So there should be a way to recover the luks volume file (most of it) in one HDD, and also likewise, in the other HDD.
I have had some experience thie ext3grep, extundelete and some other tools, but I'm not in the clear exactly yet which way to go...
What I do know, is the right filesystem recovery, or some filesystem reconstruction has to be carved back... inodes need to be restored for those exact two overwritten files... I'm not at home here really...
So if anybody has a fine suggestion, I'll be glad to consider them. And if I solve this on my own, It could end up being a possible life saver for some newbie somewhere sometime... Not certain that I will though...
If these were smaller HDDs I would disk dump an image of each, however these are both 3TB HDDs, I have no place to store them easily (not in one place). So I'll probably be working on the disks themselves.
Don't know if any more is needed to tell for this recovery question. What I know is that it's morning soon here in Europe, and I will have to be taking some rest, if I will be able to sleep.
Last edited by miroR on Sat Mar 28, 2015 10:36 pm; edited 1 time in total |
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3131
|
| Posted: Wed Nov 12, 2014 8:09 pm Post subject: |
|
|
I'm afraid whatever has been already overriten can't be easily restored. DD not only damaged filesystem (where you could try some recovery software), but also files themselves. Recovering files you have but don't know where is one thing, recovering files no-longer-there is quite another.
At this point there seems to be literaly one possible solution: restore from backup. If you are still able to mount those volumes (readonly mode?) you might try to copy the rest. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Wed Nov 12, 2014 8:48 pm Post subject: |
|
|
miroR,
Lets ignore the LUKS encrypted container issue for a moment.
You have overwritten the start of one filesystem in a file with the start of another filesystem in a file.
Unfortunately most of the metadata is at the start of the filesystem, so the data pointing to the data you want to recover is gone.
Depending on how far dd got, more or less of the data is gone too.
If the filesystem in the file is extX you might try mounting it with an alternate superblock, past the overwfitten area. I don't hold out much hope for this an the filesystem is a filesystem of two halves and is therefore corrupt. You may be able to poke about with a hex editor or Sluthkit.
All of the above assumes you can unlock the LUKS container.
That you have a luks header backup may be useful and may be another problem.
e.g. You may only be able to unlock (unscramble) the new part or the old part of the damaged filesystem at any one time.
How much data do you need back?
Before you do anything else, make some working copies (several) that you can afford to trash.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Thu Nov 13, 2014 3:53 pm Post subject: |
|
|
Really thank you for looking into my issue, szatox and NeddySeagoon.
I am still downcast and fearful of maybe having to accept the possible huge loss, on top of not good health, and other problems... So I haven't done a single move since I posted yesterday in this topic. And I haven't even been online. I'm trembling a little even at this time... But I'll stand whatever will happen fine.
First let me try to describe in more detail how I made the filesystems in question, in the two luks volumes, after I created them.
I do it generally this way (it applies for both volumes above, of course one of them would be, say, loop1 and H_E09-X instead; also the creating of the zeroes file from /dev/zero is described in the first post, this is further descriptions on top of those):
| Code: |
# losetup /dev/loop0 H_E09.vol
# cryptsetup luksFormat /dev/loop0
...
[ here luks asks for a password and I give one, and re-type it ]
...
# cryptsetup luksOpen /dev/loop0 H_E09
...
[ here luks asks and accepts the password that I gave before ]
...
|
I described all that (from memory), to tell what filesystem is there:
| Code: |
# mke2fs -t ext4 /dev/mapper/H_E09
...
[ here the mke2fs creates the filesystem, verbosely as it would on any other partition or volume ]
...
|
And now I can mount it:
| Code: |
# mount /dev/mapper/H_E09 H_E09.mnt
|
Those two volumes were full of exactly the same collection of files.
And on those two filesystems on those two volumes-files, I made that horrible mistake above of creating an empty filesystem (by dd'ing /dev/zero as explained previously) over the ext4 filesystem, but see [1] in bottom, and so overwriting the old filesystem with its metadata (as Neddy says in the quote that I bring on below).
If anything is ever recovered, whatever is recovered on either is fine instead of the other's same content.
Let me think and write more as I reread what the fine senior Gentoo boy kindly wrote:
| NeddySeagoon wrote: |
You have overwritten the start of one filesystem in a file with the start of another filesystem in a file.
Unfortunately most of the metadata is at the start of the filesystem, so the data pointing to the data you want to recover is gone.
|
I guess the metadata which is, as you clearly explain, at the start of the filesystem, is pretty certainly overwritten with zeroes, then, in it's entirety... ? Because...
| NeddySeagoon wrote: |
Depending on how far dd got, more or less of the data is gone too.
|
...Because the dd got deep into the beginning of the file (in case it went exactly over the previous file... Did it? How do I investigate whether it did... Sure, to learn some hex editor and see what Sluthkit is and what it can do for me --sure that's likely next in my plans, or at least soon.
| Code: |
-rw-r--r-- 1 root root 1670819840 2014-11-11 23:21 H_E09.vol
-rw-r--r-- 1 root root 4724903936 2014-11-11 23:21 H_E09.vol
|
while the whole file on both HDDs were previously worth:
| Code: |
-rw-r--r-- 1 root root 465567744000 2014-09-11 23:07 H_E09.vol
|
I hope the overwriting did not necessarily go linear, but...
But I can't tell, because I have never yet known anything really about filesystem internals. I guess I might have to teach myself a crash course from wherever I can find the information for me (FOSS Linux is my passion, and great resources still in our still pretty free niche in the computing world).
So let me think further.
( It is clear to me that this below applies only if I unlock the LUKS container. And I explained above in this post that the filesystem is ext4.)
| NeddySeagoon wrote: |
If the filesystem in the file is extX you might try mounting it with an alternate superblock, past the overwfitten area. I don't hold out much hope for this an the filesystem is a filesystem of two halves and is therefore corrupt. You may be able to poke about with a hex editor or Sluthkit.
|
I only partly understand the above. The notion of two halves is almost completely unclear to me, while creating (?) an alternate superblock, and mounting the filesystem with that alternate superblock, is little or no clearer either.
| NeddySeagoon wrote: |
That you have a luks header backup may be useful and may be another problem.
|
How so? I thought not having the header backup would mean that I would not be able to recover nothing at all, because I thought the dd overwrites the header first (but my knowledge is very short here as I said already).
| NeddySeagoon wrote: |
e.g. You may only be able to unlock (unscramble) the new part or the old part of the damaged filesystem at any one time.
|
Unclear, yet (crash course pending).
| NeddySeagoon wrote: |
Before you do anything else, make some working copies (several) that you can afford to trash.
|
One of the reasons that I am late, or more precisely slow with my reply (I generally don't work fast at all, I'm a late adopter, and am 57 so not fresh in adopting), among other things, is that I need to make those copies... And I don't have the room in one place for such huge data. Namely, it's 3TB partition, two of them, that I would need to dd --one suffices as explained-- somewhere, and I don't have the moneys for a bigger HDD that could hold those dumps (or the dump --one single dump for a single partition is best, else far more complex then I can manage). I may even have to think about borrowing to buy that one 4TB HDD, to be able to dd one of the two partitions.
Because those luks files-volumes were created in a single partition comprising all that gdisk got in two different 3TB HDDs.
I'll be thinking more on what to try next. The crash course will take all of my available time, but that is not more than an equivalent of part time working days (because of other issues in my life).
Returning to the issue of what program to use, I also have a few qualms left whether to try what worked for me quite some time ago, or some of the similar programs. ext3grep worked for me, but I found it wasn't much developed this of last year, and maybe extundelete or some other program could help here...
I know those programs search and find the superblock and inodes and stuff...
I'm hesitant whether borrow money to copy the partitions in two different places each --actually one suffices. I know how to do that, to the bit. I understand that much about computing, but it's quite a number of hours dumping 3TB... It would be a nightmare if I needed to go splitting and moving around the initial splits before thay fill in the entire smaller partition... I would not know a command to split a creating dd dump into different places... Or maybe I would! By using symlinks maybe... Not sure.
Any ideas? 3TB partition to dump into two different storage spaces, such as two different HDDs, one 2TB the other 1TB (and if it don't fit, yet another 1TB HDD)...
As I said, I can only be thinking, I can't decide quickly here. Too complex the issue.
[1] The filename did get overwritten, that is obvious, but, on furher musing, it's a new zeroes only file... The luks volume that was formatted and used in the previous file of the same name, wasn't mounted at the time of my creating of the new file with the same name as the, then and by virtue of that creating of mine, overwritten file, but what exactly was overwritten. The name and what more?
EDIT START (two hours later): All this musing under [1] is wrong, because you can not undelete what is altered, and not deleted (removed inodes of)... My horrible mistake consisted in altering the existing file H_E09.vol and altering the very start of it, filling in the first 1.5GB/4.5GB respectively, with zeroes... This is not where either ext3grep or extundelete can help, in the least... Somebody confirm this latest understanding of mine if they find it correct, and sorry for wandering in my quest...
EDIT END
With the information that I have given here (and the disclaimer about my insufficient knowledge of filesystems), it may be that those, in human-readable sizes, 1.5GB and 4.5GB lines of zeros have _not_ been written anywhere important really, and so, that the recovery should be possible, of the luks volume-file.
Ah, the crash course that I have to give me, from unknown resources in our FOSS *nix world... I'd need to know how the luks volume is created, is the header created in the very start of the luks volume being created, as a lay user would expect by the name "header"?
And, how are files in ext4 created? I hope developers didn't devise it in such way as to overwrite exactly the files that were just deleted, as the use of ext4grep showed to me that files can often be found and restored.
However, it's that exact one file to recover... The inode if it can be found... Getting lost... |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Thu Nov 13, 2014 9:03 pm Post subject: |
|
|
miroR,
When you make an extX filesystem, you get several copies of the superblock. Normally the one at the start of the filesystem is used and you think nothing of it.
When this superblock is damaged, mount fails, but ...
| man mount: | sb=n Instead of block 1, use block n as superblock. This could be
useful when the filesystem has been damaged. (Earlier, copies
of the superblock would be made every 8192 blocks: in block 1,
8193, 16385, ... (and one got thousands of copies on a big
filesystem). Since version 1.08, mke2fs has a -s (sparse
superblock) option to reduce the number of backup superblocks,
and since version 1.15 this is the default. Note that this may
mean that ext2 filesystems created by a recent mke2fs cannot be
mounted r/w under Linux 2.0.*.) The block number here uses 1 k
units. Thus, if you want to use logical block 32768 on a
filesystem with 4 k blocks, use "sb=131072". |
The bad news is that this does not recover all of the metadata.
If you watch the output of mke2fs, it tells the superblock locations in terms of filesystem blocks. You will need to convert that to 1k blocks.
dd carefully overwrote your old LUKS volumes as far as | Code: | -rw-r--r-- 1 root root 1670819840 2014-11-11 23:21 H_E09.vol
-rw-r--r-- 1 root root 4724903936 2014-11-11 23:21 H_E09.vol |
then truncated the files. The filesystem marked the rest of the space free, so we don't know whick blocks it occupies, nor in which order to put the blocks together to reconstruct the LUKS container. Normally, you can get some clues frem the content of the free space but with LUKS in use, the free space will look like random data. The container has to be reconstructed before it can be unlocked to reveal your data.
I do not use LUKS but my limited knowledge suggests that each instance will use a different encrytion key. Thus as it in now you can open the zeroed truncated volumes but the remaing data will use they key from the old header, so will remain scrambled.
In the event that you can reconstruct the old LUKS container and restore its header, the key in the header will decrypt the old data but you will not get zeros back from the overwritten fragment due to the key change.
I see one glimmer of hope and it depends a lot on luck. We can determine where your original file started as the new truncated file starts in the same place.
We know how long the original LUKS container was. We can assume (this is where luck comes in) the blocks were allocated sequentially.
You can create a new file with dd by copying the right number of blocks from the right start.. We will assume that this is your lost LUKS container
There are several problems with this. A 430G file is a huge file and extX is likely to put several copies of metadata in this 430G of space. You only want filesystem data blocks but dd can't tell the difference. It works beneth the filesystem. Lets come back to that.
With your 430G file, that is almost but not quite what you want, you can restore the original LUKS header and attempt to unlock it.
There are lots of ways this can go wrong. You are almost certain to pick up underlying filesystem backup metadata that you don't want.
If you can determine where it is, it can be removed.
Data blocks may not have been allocated seqentially.
When you do all this, you may not be able to open the LUKS volume anyway.
dd has some useful options
| man dd: |
seek=N skip N obs-sized blocks at start of output
skip=N skip N ibs-sized blocks at start of input
|
Its probably worth looking at the host filesystem, or better yet, an image of if, with a forensics tool before you invest a lot of time playing about with low level tools.
There way well be a lot more metadata left behind on the host filesystem that I don't know about. Sleuthkit is one such tool. There are others.
This is the data you need to reconstruct the LUKS container.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Thu Nov 13, 2014 10:28 pm Post subject: |
|
|
| NeddySeagoon wrote: |
...
[ Lots of very useful tips, will carefully read a few more times ]
...
|
and this:
| NeddySeagoon wrote: |
Its probably worth looking at the host filesystem, or better yet, an image of if,
|
Absolutely, so will soon almost be nearing hunger, to be able to get the storage necessary for the entire disk partition, the host of the luks file-volume... Unless I find a better way after I iresearch more based on your tips...
| NeddySeagoon wrote: |
with a forensics tool before you invest a lot of time playing about with low level tools.
There way well be a lot more metadata left behind on the host filesystem that I don't know about. Sleuthkit is one such tool. There are others.
This is the data you need to reconstruct the LUKS container. |
So Sleuthkit documentation and whatever I'll be pointed at from there.
Thanks a lot, NeddySeagoon, especially for bearing with my lack of and slowliness in understanding! |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sat Mar 28, 2015 9:27 pm Post subject: |
|
|
=============
This and the next post or so is fruit of maybe a week of my work before logging here at all. Clumsy, but it's serious effort of mine.
=============
This is what I'm trying to do.
This is the HDD that I got some 430GB luks volume in, that I will, hopefully, recover. (Surely, I've changed the serials in the entire topic. Other data is real.)
| Code: |
uabox ~ # smartctl -i /dev/sdc | egrep 'Serial|Capacity'
Serial Number: WD-WCC080840563
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
uabox ~ #
|
And, since I don't have a larger HDD (I wished for 4TB disk for comfort, but the regime has very much impoverished me and they keep more costs still coming at me)...
I'll try and clone that one entire HDD above, onto this 3TB one:
| Code: |
uabox ~ # smartctl -d sat -i /dev/sdb | egrep 'Serial|Capacity'
Serial Number: WD-WCC080676993
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
uabox ~ #
|
I guess the command that I need to run is:
| Code: |
uabox ~ # dd if=/dev/sdc of=/dev/sdb
|
No, I'm not exactly completely certain, but almost completely certain. So much that I think I should just run it.
It'll take really long, since both the HDDs are on two different SATA-USB3 adaptors, which mostly work (when I kindly or otherwise, talk them into working).
| Code: |
uabox ~ # dd if=/dev/sdc of=/dev/sdb &
[1] 3664
uabox ~ #
|
That great unix command runs quietly.
I'm really doing some other work:
| Code: |
Tasks: 182 total, 3 running, 179 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.6 us, 6.2 sy, 65.6 ni, 5.6 id, 4.3 wa, 0.0 hi, 0.7 si, 0.0 st
KiB Mem : 16385720 total, 6129416 free, 604184 used, 9652120 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15694152 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3610 miro 20 0 702012 165564 20156 S 492.1 1.0 120:13.27 ffmpeg
3664 root 20 0 9120 828 748 R 25.2 0.0 1:16.24 dd
3555 root 20 0 0 0 0 R 13.2 0.0 0:40.15 usb-storage
2974 root 20 0 175488 30020 16776 S 1.7 0.2 0:19.72 X
12 root 20 0 0 0 0 S 1.0 0.0 0:02.38 ksoftirqd/1
16 root 20 0 0 0 0 S 1.0 0.0 0:02.47 ksoftirqd/2
28 root 20 0 0 0 0 S 1.0 0.0 0:02.78 ksof
|
(
a digression, a happy one. This:
Libav (Avconv) Imposition on Users who want FFmpeg
https://forums.gentoo.org/viewtopic-t-989196-highlight-.html?sid=d26312b83d824ecf4d4f18cf80bb0053
has somewhat been a success. Relatively recently --go and see the bug, and links there, I don't have time, this and other issues have been making me struggle really hard-- ffmpeg is not anymore treated badly, but is an easier choice to acquire for beginners, does not depend on having to support its regression (as mv said), the libav, it was in the news, type `eselect news list' and find it...
)
But I'll post here what `man dd' says, and which is necessary in long hauls like this one
| Code: |
Sending a USR1 signal to a running 'dd' process makes it print I/O statistics to
standard error and then resume copying.
$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid; sleep 1; kill $pid
18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB) copied,
34.6279 seconds, 271 MB/s
|
Sure, I could have included the string `& pid=$!' at the end of the command that is currently running, instead of the last `&', but:
| Code: |
uabox ~ # ps aux | grep [d]d
...[snip]...
root 3664 25.7 0.0 9120 828 pts/5 R 20:17 2:52 dd if=/dev/sdc of=/dev/sdb
uabox ~ #
|
gave me back the pid number: 3664. So I can:
| Code: |
uabox ~ # kill -USR1 3664
14039841+0 records in
14039841+0 records out
7188398592 bytes (7.2 GB) copied, 772.552 s, 9.3 MB/s
uabox ~ #
|
... Ouch! [So I can  ] see that it's too slow! Similar kind of dd'ing usually run faster... at around 40 MB/s, sometimes even 90 MB/s...
This slowliness is not acceptable. Oh, no! It is... Sorry! It's no processing power left for better (see the top output above)... Whoah! (EDIT later: neither was this correct assumption, read on.)
Some half or one hour later:
| Code: |
uabox ~ # kill -USR1 3664
53985297+0 records in
53985297+0 records out
27640472064 bytes (28 GB) copied, 3285.1 s, 8.4 MB/s
uabox ~ #
|
So, first I killed the ffmpeg process, and also:
| Code: |
uabox ~ # jobs
[1]+ Running dd if=/dev/sdc of=/dev/sdb &
uabox ~ # kill -9 %1
uabox ~ #
|
which later returned notice that the dd process was terminated.
Lets try now:
| Code: |
uabox ~ # dd if=/dev/sdc of=/dev/sdb &
[1] 3961
uabox ~ # touch started
uabox ~ # ls -l started
-rw-r--r-- 1 root root 0 2015-03-25 21:15 started
uabox ~ #
|
| Code: |
uabox ~ # kill -USR1 3961
1630530+0 records in
1630529+0 records out
834830848 bytes (835 MB) copied, 66.0158 s, 12.6 MB/s
uabox ~ #
|
Still too slow.
Restarted the machine, thinking maybe that'll help, but no. Something probably like the initial areas of the HDD, where the partition is, is different then the rest, and can't be treated as the rest, maybe because of the block size, or whatever the names of those.
When restarted, also removed other devices, so only these two SATA-to-USB3 adaptors are attached with these 2 HDDs of 3TB equal capacity and model.
And when restarted, decided to go for the /dev/sdc1 and /dev/sdb1 dumping one another to (naming changed on restart, so describing it in that fashion).
And it's going on fine. This is the command I run:
| Code: |
uabox ~ # run_CMD.sh 1000 60 "kill -USR1 3123"
|
The command is nothing much, but useful for repetitive listing like these, and I keep it in my /usr/local/bin :
run_CMD.sh:
| Code: |
#!/bin/bash
LIMIT=$1
for ((i=1; i<=LIMIT; i++))
do
echo $i of $LIMIT at $2s int.
$3
sleep $2
echo " "
echo " "
echo "------------------------------------------------------------"
done
exit=0
|
The output, at:
| Code: |
uabox ~ # date
Wed 25 Mar 22:37:52 CET 2015
uabox ~ #
|
and the `started' (which I touched right when I issued the dd command anew) has timestamp:
| Code: |
uabox ~ # ls -l started
-rw-r--r-- 1 root root 0 2015-03-25 21:40 started
uabox ~ #
|
So the output is:
| Code: |
54 of 1000 at 60s int.
511342498+0 records in
511342498+0 records out
261807358976 bytes (262 GB) copied, 3247.59 s, 80.6 MB/s
------------------------------------------------------------
55 of 1000 at 60s int.
517020129+0 records in
517020128+0 records out
264714305536 bytes (265 GB) copied, 3307.59 s, 80.0 MB/s
------------------------------------------------------------
56 of 1000 at 60s int.
522677982+0 records in
522677981+0 records out
267611126272 bytes (268 GB) copied, 3367.61 s, 79.5 MB/s
uabox ~ # date
Wed 25 Mar 22:37:52 CET 2015
uabox ~ #
------------------------------------------------------------
57 of 1000 at 60s int.
528376202+0 records in
528376202+0 records out
270528615424 bytes (271 GB) copied, 3427.62 s, 78.9 MB/s
------------------------------------------------------------
58 of 1000 at 60s int.
533985952+0 records in
533985952+0 records out
273400807424 bytes (273 GB) copied, 3487.62 s, 78.4 MB/s
|
And the `top', after --seeing how the rates were sustained-- firing up the ffmpeg processiong again:
| Code: |
top - 22:42:19 up 1:11, 2 users, load average: 9.90, 9.98, 7.75
Tasks: 175 total, 5 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu(s): 14.9 us, 19.8 sy, 56.5 ni, 4.0 id, 1.1 wa, 0.0 hi, 3.7 si, 0.0 st
KiB Mem : 16385720 total, 88068 free, 597240 used, 15700412 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15696288 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3231 miro 20 0 701904 147776 3380 S 407.6 0.9 70:10.06 ffmpeg
3123 root 20 0 9116 1176 1072 R 85.8 0.0 59:27.27 dd
34 root 20 0 0 0 0 D 26.8 0.0 13:52.30 kworker/u1+
842 root 20 0 0 0 0 S 12.3 0.0 10:47.23 kswapd0
...[snip]...
|
And that's not too slow.
I'll let it do its work now.
---
More than half a day later.
This is how it went.
| Code: |
------------------------------------------------------------
995 of 1000 at 60s int.
5802572001+0 records in
5802572000+0 records out
2970916864000 bytes (3.0 TB) copied, 59738.9 s, 49.7 MB/s
------------------------------------------------------------
996 of 1000 at 60s int.
5808238553+0 records in
5808238553+0 records out
2973818139136 bytes (3.0 TB) copied, 59799 s, 49.7 MB/s
------------------------------------------------------------
997 of 1000 at 60s int.
5813887305+0 records in
5813887305+0 records out
2976710300160 bytes (3.0 TB) copied, 59859 s, 49.7 MB/s
------------------------------------------------------------
998 of 1000 at 60s int.
5819571465+0 records in
5819571465+0 records out
2979620590080 bytes (3.0 TB) copied, 59919.1 s, 49.7 MB/s
------------------------------------------------------------
999 of 1000 at 60s int.
5825253239+0 records in
5825253239+0 records out
2982529658368 bytes (3.0 TB) copied, 59979.2 s, 49.7 MB/s
uabox ~ # ls -l started
-rw-r--r-- 1 root root 0 2015-03-25 21:40 started
uabox ~ # date
Thu 26 Mar 14:20:41 CET 2015
uabox ~ #
------------------------------------------------------------
1000 of 1000 at 60s int.
5830950144+0 records in
5830950143+0 records out
2985446473216 bytes (3.0 TB) copied, 60039.3 s, 49.7 MB/s
uabox ~ # jobs
[1]- Running dd if=/dev/sdc1 of=/dev/sdb1 &
[2]+ Running run_CMD.sh 1000 60 "kill -USR1 3123" &
------------------------------------------------------------
uabox ~ #
[2]+ Done run_CMD.sh 1000 60 "kill -USR1 3123"
uabox ~ #
uabox ~ # date
Thu 26 Mar 14:26:35 CET 2015
uabox ~ #
5860531087+0 records in
5860531087+0 records out
3000591916544 bytes (3.0 TB) copied, 60403.8 s, 49.7 MB/s
[1]- Done dd if=/dev/sdc1 of=/dev/sdb1
uabox ~ # date
Thu 26 Mar 14:27:56 CET 2015
uabox ~ #
|
I described the problem, that I intend to --hopefully-- solve these days, before in this topic.
The first thing next, is to --may it take as long as it needs, but... But the first thing is firing up the Autopsy browser and letting the Sleuth, the kit called Sleuthkit, calculate the MD5 sum. Even if I first have to anyhow end up with a modified filesystem, and not this exact one, but modified version of this one, after the luks header restoration. I still want to have the Sleuth take the MD5 sum of this unmodified dump.
Why? Because during that time, I'll read about restoring the luks header, and then, when I do try to restore it, if I happen to go wrong, I'll be able to retrace my steps, and end up exactly at this same stage right here where you're reading now, and verify the sum. The sum will have to be the same, if the partition that I will have dumped for the second time (this is the first) hasn't been modified in any way, and it won't be. A huge number of bits all in one same exact order! That's computing.
============== MAYBE MOVE THIS UP, START =========
Maybe first check the dump that I made.
| Code: |
uabox ~ # smartctl -d sat -i /dev/sdb
...[snip]...
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00AZ6B0
Serial Number: WD-WCC080676993
LU WWN Device Id: 5 0014ee 2b2f50074
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Thu Mar 26 16:07:42 2015 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
|
I don't know if I needed to do this, but it should not hurt.
| Code: |
uabox ~ # blockdev --setro /dev/sdb
uabox ~ # blockdev --getro /dev/sdb
1
uabox ~ #
|
| Code: |
uabox ~ # mkdir /mnt/sdk1
uabox ~ # blockdev --getro /dev/sdb1
0
uabox ~ # blockdev --setro /dev/sdb1
uabox ~ # blockdev --getro /dev/sdb1
1
uabox ~ # mount -r /dev/sdb1 /mnt/sdk1
uabox ~ # mount
...[snip]...
/dev/sdb1 on /mnt/sdk1 type ext4 (ro)
uabox ~ #
|
And this is the state of my HDD with lots of work remaining on it for me:
| Code: |
uabox ~ # ls -l /mnt/sdk1/
total 457295768
-rw-r--r-- 1 miro miro 0 2013-08-31 17:06 000_WCC080840563_THIS
-r-------- 1 root root 1052672 2014-05-06 01:03 H_E05.bak
-rw-r--r-- 1 root root 76 2014-05-06 01:04 H_E05.bak.sum
drwxr-xr-x 2 miro miro 20480 2014-05-07 16:34 H_E05_fL
-rw-r--r-- 1 miro miro 24277 2014-05-07 16:24 H_E05_fL.sum
drwxr-xr-x 2 miro miro 4096 2014-05-06 00:58 H_E05.mnt
-rw-r--r-- 1 miro miro 23512 2014-05-07 16:24 H_E05.sum
-rw-r--r-- 1 root root 465567744000 2014-09-11 23:09 H_E05.vol
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:26 H_E09.mnt
-rw-r--r-- 1 miro miro 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 2700836864 2014-11-11 23:21 H_E09.vol
-rw-r--r-- 1 miro miro 61 2014-09-11 23:11 ls_list
uabox ~ #
|
Without these:
| Code: |
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
|
nothing would be possible. I hope those are still in good shape, all bits in their place.
| Code: |
uabox ~ # blockdev --getro /dev/sdb1
1
uabox ~ # blockdev --getro /dev/sdb
1
uabox ~ # cd /mnt/sdk1/
uabox sdk1 # sha256sum -c H_E09.bak.sum
H_E09.bak: OK
uabox sdk1 #
uabox sdk1 # cd -
/root
uabox ~ #
|
( You see how I got out quickly like a coward.  I'm scared, my friend, still scared for my data... )
============== MAYBE MOVE THIS UP, END =========
I fire up Autopsy, running as root:
| Code: |
uabox ~ # autopsy &
[1] 1689
uabox ~ #
|
and then, esp. since javascript is not recommended (even though I probably won't go online yet), this is a fine choice:
| Code: |
ukra@uabox ~ $ links -g http://localhost:9999/autopsy
|
(
Another short digression:
I'm happily using links for autopsy, instead of the SchmoogleFox --yes I mean the Google-infested Firefox--, thanks to jonathan183's suggestion here:
Updating and keeping your Gentoo non-poetterized
https://forums.gentoo.org/viewtopic-t-1012022-start-25.html#7713692
--just pls. read the topic and redirect where I kindly ask it you want to go about browsers...--
But I don't graphical links for posting on Gentoo forums, because SSL don't work. SSL does work with dillo, and dillo is great for posting on Gentoo Forums!
)
And I endulge (yet again; I managed to recover, that was almost for practice, and out of laziness to rewrite all --which would be faster-- , exactly what I wrote and lost the draft of; some 4-5 hours of work worth of text; plain text, a few days ago, so I'm getting confident, but am not yet there)...
And I endulge in more reading of the Help where the fine creators of Sleuth gave us all the necessary documentation.
Ah, I remember what I need to do. Replace a few entries in the /etc/autopsy.pl with some more sensible values:
This is the conf as it installs by default:
| Code: |
# Autopsy configuration settings
# when set to 1, the server will stop after it receives no
# connections for STIMEOUT seconds.
$USE_STIMEOUT = 0;
$STIMEOUT = 3600;
# number of seconds that child waits for input from client
$CTIMEOUT = 15;
# set to 1 to save the cookie value in a file (for scripting)
$SAVE_COOKIE = 1;
$INSTALLDIR = '/usr/lib/autopsy';
# System Utilities
$GREP_EXE = '/bin/grep';
$FILE_EXE = '/usr/bin/file';
$MD5_EXE = '/usr/bin/md5sum';
$SHA1_EXE = '/usr/bin/sha1sum';
# Directories
$TSKDIR = '/usr/bin/';
$NSRLDB = '';
$LOCKDIR = '/tmp';
|
And the diff for what I changed it to:
| Code: |
27c27
< $LOCKDIR = '/tmp';
---
> $LOCKDIR = '/Cmn/autopsy';
|
And you're right, it wasn't worth a diff. Never mind.
Important is, I have lots of room there, more comfortable. (I like to have most of my
data completely separated from the system, not even in /home/ukra, and that's where it is, in /Cmn, a dir on a separate partition than the system.)
While I'm not completely certain if I need to do this:
| Code: |
uabox ~ # umount /mnt/sdk1/
uabox ~ #
|
I'm almost sure it won't hurt.
No, nothing there.
But this:
| Code: |
uabox ~ # blockdev --getro /dev/sdb
1
uabox ~ # blockdev --getro /dev/sdb1
1
uabox ~ #
|
shouldn't hurt to remain.
---
And I opened a new case which I named WCC070-luks-vol using Links running the Autopsy browser.
It looks like this:
| Code: |
uabox ~ # ls -lR /Cmn/autopsy/WCC070-luks-vol/
/Cmn/autopsy/WCC070-luks-vol/:
total 16
-rw-r--r-- 1 root root 138 2015-03-26 17:34 case.aut
-rw-r--r-- 1 root root 167 2015-03-26 17:35 case.log
-rw-r--r-- 1 root root 10 2015-03-26 17:34 investigators.txt
drwxr-xr-x 7 root root 4096 2015-03-26 17:35 ukrabox
/Cmn/autopsy/WCC070-luks-vol/ukrabox:
total 24
-rw-r--r-- 1 root root 115 2015-03-26 17:35 host.aut
drwxr-xr-x 2 root root 4096 2015-03-26 17:35 images
drwxr-xr-x 2 root root 4096 2015-03-26 17:38 logs
drwxr-xr-x 2 root root 4096 2015-03-26 17:35 mnt
drwxr-xr-x 2 root root 4096 2015-03-26 17:35 output
drwxr-xr-x 2 root root 4096 2015-03-26 17:35 reports
/Cmn/autopsy/WCC070-luks-vol/ukrabox/images:
total 0
/Cmn/autopsy/WCC070-luks-vol/ukrabox/logs:
total 12
-rw-r--r-- 1 root root 128 2015-03-26 17:35 host.log
-rw-r--r-- 1 root root 273 2015-03-26 17:39 miroRovis.exec.log
-rw-r--r-- 1 root root 46 2015-03-26 17:35 miroRovis.log
/Cmn/autopsy/WCC070-luks-vol/ukrabox/mnt:
total 0
/Cmn/autopsy/WCC070-luks-vol/ukrabox/output:
total 0
/Cmn/autopsy/WCC070-luks-vol/ukrabox/reports:
total 0
uabox ~ #
|
Some more, and it will be complete information for now:
| Code: |
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/case.aut
# Autopsy case config file
# Case: WCC070-luks-vol
created Thu Mar 26 17:34:38 2015
images images
data output
log logs
reports reports
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/case.log
Thu Mar 26 17:34:38 2015: Case WCC070-luks-vol created
Thu Mar 26 17:35:20 2015: Host ukrabox added to case
Thu Mar 26 17:35:23 2015: Host ukrabox opened by miroRovis
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/investigators.txt
miroRovis
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/host.aut
# Autopsy host config file
# Case: WCC070-luks-vol Host: ukrabox
# Created: Thu Mar 26 17:35:20 2015
timeskew 0
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/logs/host.log
Thu Mar 26 17:35:20 2015: Host ukrabox added to case WCC070-luks-vol
Thu Mar 26 17:35:23 2015: Host ukrabox opened by miroRovis
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/logs/miroRovis.exec.log
Thu Mar 26 17:38:19 2015: '/usr/bin/img_stat' -t "/dev/sdb1"
Thu Mar 26 17:38:19 2015: '/usr/bin/fsstat' -t -i raw "/dev/sdb1"
Thu Mar 26 17:39:52 2015: '/usr/bin/img_stat' -t "/dev/sdb1"
Thu Mar 26 17:39:52 2015: '/usr/bin/blkls' -f raw -e "/dev/sdb1" | '/usr/bin/md5sum'
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/logs/miroRovis.log
Thu Mar 26 17:35:23 2015: Host ukrabox opened
|
where the very last, and for long hours next to be the only thing being done is:
| Code: |
Thu Mar 26 17:39:52 2015: '/usr/bin/blkls' -f raw -e "/dev/sdb1" | '/usr/bin/md5sum'
|
( When I started the calculating of that sum, the browser simply opened a new page with the notice:
| Autopsy wrote: |
Calculating MD5 (this could take a while)
|
and I'll only be seeing that notice as far as Autopsy goes, for many hours next. )
Not much of a strain on the system either:
| Code: |
top - 17:55:51 up 20:24, 2 users, load average: 11.20, 10.84, 9.77
Tasks: 184 total, 2 running, 182 sleeping, 0 stopped, 0 zombie
%Cpu(s): 22.8 us, 8.6 sy, 61.2 ni, 4.0 id, 2.9 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 16385720 total, 137484 free, 629932 used, 15618304 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15665580 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1931 miro 20 0 702368 164876 19344 S 460.1 1.0 66:28.86 ffmpeg
1982 root 20 0 9100 740 664 S 20.8 0.0 1:11.68 md5sum
1981 root 20 0 29908 3212 2832 D 20.1 0.0 1:09.22 blkls
1916 root 20 0 29908 3208 2832 D 19.1 0.0 5:54.48 blkls
1917 root 20 0 9100 740 664 S 17.2 0.0 5:49.94 md5sum
1148 root 20 0 0 0 0 D 7.6 0.0 31:25.12 usb-storage
...[snip]...
|
so I can use my time for other work.
I just (Thu 26 Mar 18:48:22 CET 2015) noticed I got a "Receive timeout" showing in bottom of Links, and the page shows this error (which is not copy-paste-able so this is manual typing:
| Code: |
================== Error ===================
Error loading
http://localhost:9999/autopsy?<a long cgi-variables string here>
Receive timeout
================== Cancel ===================
|
where Error and Cancel appear to be links or buttons, but all is currently frozen, apparently.
However, since, if I fire `top' again:
| Code: |
top - 18:54:20 up 21:23, 2 users, load average: 12.54, 12.33, 12.24
Tasks: 190 total, 3 running, 187 sleeping, 0 stopped, 0 zombie
%Cpu(s): 20.2 us, 7.8 sy, 66.6 ni, 2.2 id, 2.8 wa, 0.0 hi, 0.6 si, 0.0 st
KiB Mem : 16385720 total, 96892 free, 635224 used, 15653604 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15660468 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1931 miro 20 0 702136 164968 19344 S 476.3 1.0 340:54.97 ffmpeg
1982 root 20 0 9100 740 664 S 13.5 0.0 9:02.97 md5sum
1981 root 20 0 29908 3212 2832 D 13.2 0.0 8:47.29 blkls
2031 root 20 0 9104 744 664 R 12.2 0.0 7:02.40 md5sum
1916 root 20 0 29908 3208 2832 D 11.5 0.0 13:23.16 blkls
2030 root 20 0 29908 3212 2832 R 11.5 0.0 6:47.50 blkls
1917 root 20 0 9100 740 664 S 11.2 0.0 13:31.67 md5sum
1148 root 20 0 0 0 0 D 6.9 0.0 35:48.10 usb-storage
...[snip]...
|
and the
| Code: |
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/logs/miroRovis.exec.log
|
above reported now has:
| Code: |
Thu Mar 26 17:49:52 2015: '/usr/bin/img_stat' -t "/dev/sdb1"
Thu Mar 26 17:49:53 2015: '/usr/bin/blkls' -f raw -e "/dev/sdb1" | '/usr/bin/md5sum'
Thu Mar 26 17:59:53 2015: '/usr/bin/img_stat' -t "/dev/sdb1"
Thu Mar 26 17:59:53 2015: '/usr/bin/blkls' -f raw -e "/dev/sdb1" | '/usr/bin/md5sum'
|
those lines added, all may still be fine.
Anyway, the Autopsy is just the frontend. These commands that are running, and they are part of the SleuthKit, is what matters. I guess I'll get the sum calculated just the same, and then I can restart the Autopsy browser and work on the case further. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sat Mar 28, 2015 11:01 pm Post subject: |
|
|
So let's see what cryptsetup' says about header backup and restore and other that might be relevant in my
case.
| Code: |
...[snip]...
WARNINGS
...[snip]...
LUKS header: If the header of a LUKS volume gets damaged, all data is permanently lost unless you
have a header-backup. If a key-slot is damaged, it can only be restored from a header-backup or if
another active key-slot with known passphrase is undamaged. Damaging the LUKS header is something
people manage to do with surprising frequency. This risk is the result of a trade-off between
security and safety, as LUKS is designed for fast and secure wiping by just overwriting header and
key-slot area.
...[snip]...
LUKS EXTENSION
...[snip]...
luksHeaderBackup <device> --header-backup-file <file>
Stores a binary backup of the LUKS header and keyslot area.
Note: Using '-' as filename writes the header backup to a file named '-'.
WARNING: This backup file and a passphrase valid at the time of backup allows decryption
of the LUKS data area, even if the passphrase was later changed or removed from the LUKS
device. Also note that with a header backup you lose the ability to securely wipe the LUKS
device by just overwriting the header and key-slots. You either need to securely erase all
header backups in addition or overwrite the encrypted data area as well. The second option
is less secure, as some sectors can survive, e.g. due to defect management.
luksHeaderRestore <device> --header-backup-file <file>
Restores a binary backup of the LUKS header and keyslot area from the specified file.
Note: Using '-' as filename reads the header backup from a file named '-'.
WARNING: Header and keyslots will be replaced, only the passphrases from the backup will
work afterwards.
This command requires that the master key size and data offset of the LUKS header already
on the device and of the header backup match. Alternatively, if there is no LUKS header on
the device, the backup will also be written to it.
...[snip]...
MISCELLANEOUS
repair <device>
Tries to repair the device metadata if possible. Currently supported only for LUKS device
type.
This command is useful to fix some known benign LUKS metadata header corruptions. Only
basic corruptions of unused keyslot are fixable. This command will only change the LUKS
header, not any key-slot data.
WARNING: Always create a binary backup of the original header before calling this command.
OPTIONS
...[snip]...
--align-payload <number of 512 byte sectors>
Align payload at a boundary of value 512-byte sectors. This option is relevant for
luksFormat.
If not specified, cryptsetup tries to use the topology info provided by kernel for the
underlying device to get optimal alignment. If not available (or the calculated value is a
multiple of the default) data is by default aligned to a 1MiB boundary (i.e. 2048 512-byte
sectors).
For a detached LUKS header this option specifies the offset on the data device. See also
the --header option.
...[snip]...
--header <device or file storing the LUKS header>
Use a detached (separated) metadata device or file where the LUKS header is stored. This
options allows one to store ciphertext and LUKS header on different devices.
This option is only relevant for LUKS devices and can be used with the luksFormat, open,
luksSuspend, luksResume, status and resize commands.
For luksFormat with a file name as argument to --header, it has to exist and be large
enough to contain the LUKS header. See the cryptsetup FAQ for header size calculation.
For other commands that change the LUKS header (e.g. luksAddKey), specify the device or
file with the LUKS header directly as the LUKS device.
If used with luksFormat, the --align-payload option is taken as absolute sector alignment
on ciphertext device and can be zero.
WARNING: There is no check whether the ciphertext device specified actually belongs to the
header given. In fact you can specify an arbitrary device as the ciphertext device for open
with the --header option. Use with care.
...[snip]...
NOTES ON LOOPBACK DEVICE USE
Cryptsetup is usually used directly on a block device (disk partition or LVM volume). However, if
the device argument is a file, cryptsetup tries to allocate a loopback device and map it into this
file. This mode requires Linux kernel 2.6.25 or more recent which supports the loop autoclear flag
(loop device is cleared on last close automatically). Of course, you can always map a file to a
loop-device manually. See the cryptsetup FAQ for an example.
When device mapping is active, you can see the loop backing file in the status command output.
Also see losetup(8).
...[snip]...
|
Thinking about all the above, I'm pondering to try and open the luks volume without first restoring the header. With a command like this:
| Code: |
# cryptsetup --header /mnt/sdk1/H_E09.bak open /mnt/sdk1/H_E09.vol
|
I think I can try and do it even before the calculating of the MD5 sum of the device is over.
I mounted it as I showed above.
But it's not happening instantly:
| Code: |
uabox ~ # mount -r /dev/sdb1 /mnt/sdk1/
^Z
bg
|
Will wait longer and see if it finally does.
I can see the command is being executed:
| Code: |
uabox ~ # ps aux | grep mount
root 2697 0.0 0.0 15352 720 ? Ss Mar25 0:00 /usr/sbin/rpc.mountd
root 2928 7.0 0.0 15876 2764 pts/2 D+ 23:00 0:52 mount -r /dev/sdb1 /mnt/sdk1/
root 2943 0.0 0.0 11588 2028 pts/5 S+ 23:12 0:00 grep --colour=auto mount
|
Patience.
Hmmh... I'm not so advanced to be at ease trying to background those Sleuthkit's own blkls and img_stat commands that are running.
But I can try and background the ffmpeg jobs... Did it. But no. Those don't influence the mounting, I don't think.
More patience.
Trying to kill it, and maybe issue it again:
| Code: |
uabox ~ # kill -9 2928 &
[1] 3030
uabox ~ #
[1]+ Done kill -9 2928
uabox ~ # ps aux | grep mount
root 2697 0.0 0.0 15352 720 ? Ss Mar25 0:00 /usr/sbin/rpc.mountd
root 2928 5.8 0.0 15876 2764 pts/2 D+ 23:00 2:12 mount -r /dev/sdb1 /mnt/sdk1/
root 3032 0.0 0.0 11588 2020 pts/5 S+ 23:38 0:00 grep --colour=auto mount
uabox ~ #
|
Not happening... And top isn't showing it as zombie.
And there is no way that I know of, to find out at what stage the MD5 calculation is at a particular time during its execution...
I killed it (but it did not happen immediately either), and I'm retrying now:
( this is from the same terminal, with the kill, and now the new attempt )
| Code: |
^Z^Z
^Z^Z^Z
^C^C
^C^C^C
Killed
uabox ~ #
uabox ~ # ^C
uabox ~ # mount -r /dev/sdb1 /mnt/sdk1/ &
[2] 30430
uabox ~ #
uabox ~ # date ; jobs
Fri 27 Mar 08:35:55 CET 2015
[1]- Running autopsy &
[2]+ Running mount -r /dev/sdb1 /mnt/sdk1/ &
uabox ~ #
|
top:
| Code: |
top - 09:22:15 up 1 day, 11:51, 2 users, load average: 12.76, 12.43, 12.63
Tasks: 190 total, 4 running, 186 sleeping, 0 stopped, 0 zombie
%Cpu(s): 34.4 us, 20.0 sy, 39.1 ni, 2.8 id, 3.1 wa, 0.0 hi, 0.7 si, 0.0 st
KiB Mem : 16385720 total, 91600 free, 646696 used, 15647424 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15648692 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30391 miro 20 0 702908 147852 1644 S 279.1 0.9 1581:41 ffmpeg
2031 root 20 0 9104 80 0 S 49.2 0.0 144:32.40 md5sum
1982 root 20 0 9100 76 0 S 48.8 0.0 144:27.28 md5sum
1917 root 20 0 9100 76 0 R 48.5 0.0 144:35.60 md5sum
1916 root 20 0 29908 376 0 R 39.2 0.0 130:46.58 blkls
2030 root 20 0 29908 380 0 D 38.2 0.0 136:42.41 blkls
1981 root 20 0 29908 380 0 D 37.9 0.0 131:16.32 blkls
1148 root 20 0 0 0 0 R 9.6 0.0 97:39.83 usb-storage
...[snip]...
|
I think I may have been wrong about ffmpeg jobs not influencing the speed of MD5 sum calculation. Both of those are pocessing power intensive. MD5 calculation is not so much about reading the block device content in question, but more about calculating some algorithm on it, which needs and consumes prossessing power (less available here because of the processing power intensive occupant: the ffmpeg, esp. at start --see the pastes of my previous top commnads [
OTOH, the ffmpeg is working, by now, some 3 times more slowly than when it's the sole real worker on the system. Quitting it as soon as the current conversion is done.
] ).
At:
Fri 27 Mar 09:57:38 CET 2015
quit ffmpeg jobs. No immediate increase in %CPU by md5sum and blkls jobs show.
Also, nothing new show in:
/Cmn/autopsy/WCC070-luks-vol/ukrabox/logs/miroRovis.exec.log
But I here notice that:
| Code: |
[2]+ Done mount -r /dev/sdb1 /mnt/sdk1/
|
I now have my cloned partition mounted. And who knows if I can try that readonly open I thought of above. I think it's safe to try (no I don't know that it is, I only hope that it is safe).
| Code: |
uabox ~ # ls -l /mnt/sdk1
total 457295768
-rw-r--r-- 1 miro miro 0 2013-08-31 17:06 000_WCC080840563_THIS
-r-------- 1 root root 1052672 2014-05-06 01:03 H_E05.bak
-rw-r--r-- 1 root root 76 2014-05-06 01:04 H_E05.bak.sum
drwxr-xr-x 2 miro miro 20480 2014-05-07 16:34 H_E05_fL
-rw-r--r-- 1 miro miro 24277 2014-05-07 16:24 H_E05_fL.sum
drwxr-xr-x 2 miro miro 4096 2014-05-06 00:58 H_E05.mnt
-rw-r--r-- 1 miro miro 23512 2014-05-07 16:24 H_E05.sum
-rw-r--r-- 1 root root 465567744000 2014-09-11 23:09 H_E05.vol
-r-------- 1 root root 1052672 2014-09-11 23:11 H_E09.bak
-rw-r--r-- 1 root root 76 2014-09-11 23:11 H_E09.bak.sum
drwxr-xr-x 2 root root 4096 2014-09-11 17:26 H_E09.mnt
-rw-r--r-- 1 miro miro 35810 2014-09-11 22:37 H_E09.sum
-rw-r--r-- 1 root root 2700836864 2014-11-11 23:21 H_E09.vol
-rw-r--r-- 1 miro miro 61 2014-09-11 23:11 ls_list
uabox ~ #
|
So:
| Code: |
uabox ~ # cryptsetup --header /mnt/sdk1/H_E09.bak open /mnt/sdk1/H_E09.vol
Command requires device and mapped name as arguments.
uabox ~ #
|
---
NOTE JUST BEFORE POSTING: it was only necessary to add an argument, a name-to-be such as `H_E09' at end.
---
As you can see, I got a concise answer. Trying --verbose.
| Code: |
uabox ~ # cryptsetup --verbose --header /mnt/sdk1/H_E09.bak open /mnt/sdk1/H_E09.vol
Command requires device and mapped name as arguments.
Command failed with code 22: Command requires device and mapped name as arguments.
uabox ~ #
|
The thing is, it's a file, and it's a ruined first few hundred of MB of that file with the "dd if=/dev/zero of=/where-it-was-mounted/H_E09.vol" command. It can't even be seen as a luks device. Only it takes me time to understand any of these new aspects of this damage and recovery case of mine.
I was also thinking of another thing. I think I need to try and see if I can back up the part that I'll be overwriting. So if I fail or do something wrong in my luksHeaderBackup attempt, I can go back to this starting point, without the dumping of the original HDD partition all over.
Lord, there's few precise things to do which can obviously solve this rather quickly. See the lines containing: "Also note that with a header backup you lose the ability to securely wipe the LUKS device by just overwriting the header and key-slots." by which they mean experts can recover content lost like in this case of mine relatively easily and quickly, but I'll be poring over this for so much longer, probably.
So, to be able to keep starting at the current stage (by which I mean also the MD5 calculation done, as I presume it soon will be), I need to study and apply Autopsy's "Metadata Analysis", which is about inodes in ext4 volume in question.
Which number is the inode of the file H_E09.vol (it's still the same inode as previously, nominally the same file, albeit with the firt few hundreds MB overwritten with zeroes) and where does it start, what data units it occupies (data units I think are blocks of multiple of 512 bytes).
But first, I'll finish the previous attempt. Maybe I can try setting the H_E09.vol file on a loopback device.
| Code: |
uabox ~ # losetup -f
/dev/loop0
uabox ~ # losetup /dev/loop0 /mnt/sdk1/H_E09.vol
uabox ~ #
|
No error shown. Doesn't mean anything is accomplished, but... Let's see.
| Code: |
uabox ~ # cryptsetup --verbose --header /mnt/sdk1/H_E09.bak open /dev/loop0
Command requires device and mapped name as arguments.
Command failed with code 22: Command requires device and mapped name as arguments.
uabox ~ #
|
--
NOTE JUST BEFORE POSTING: I missed the point in earlier, somewhat different attempt above, as you saw, but here it finally dawned on me what the wokds cryptsetup said in stdout mean.
--
Uhmmh. Maybe just the mapped name as argument was missing. Let's see:
| Code: |
uabox ~ # cryptsetup --verbose --header /mnt/sdk1/H_E09.bak open /dev/loop0 H_E09
Enter passphrase for /mnt/sdk1/H_E09.vol:
Key slot 0 unlocked.
Command successful.
uabox ~ #
|
But:
| Code: |
uabox ~ # mount -r /dev/mapper/H_E09 H_E09.mnt
NTFS signature is missing.
Failed to mount '/dev/mapper/H_E09': Invalid argument
The device '/dev/mapper/H_E09' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
uabox ~ #
|
Also trying:
| Code: |
uabox ~ # mount -r -t ext4 /dev/mapper/H_E09 H_E09.mnt
mount: wrong fs type, bad option, bad superblock on /dev/mapper/H_E09,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
uabox ~ #
|
What am I missing?
However, that last output is not gibberish mumbo-jumbo anymore. That actually looks promising, and now I believe I'm closer to understanding what Neddy Seagoon was talking about...
Because this last error I have seen when it was trivial, just a really quick run of fsck on such a partition, and the ext3/ext4 partition in question was back. It happened to me many times, such as, say when the system lost power, or some other cause for minor inconsistencies for which some ext3/ext4 partition would not mount regularly.
This could now be a matter of recovering the EXT4 partition in question, and I get back most of my data (just not those that were overwritten with zeroes, which was not much of all the 430G at all).
I don't think I can run e2fsck on this loop device, or can I? Yes, I think I should be able to run some readonly check, I'll go and see the manual of it.
Just, for completeness, as far as "useful info is found in syslog", this:
| Code: |
Mar 27 11:24:16 uabox kernel: [136308.798208] EXT4-fs (dm-3): VFS: Can't find ext4 filesystem
|
is all that I have in /var/log/messages about this event
Also just this line in bottom of dmesg output:
| Code: |
[136308.798208] EXT4-fs (dm-3): VFS: Can't find ext4 filesystem
|
And the `man e2fsck' tells what Neddy said. I'll post excerpts.
| Code: |
...[snip]...
OPTIONS
...[snip]...
-b superblock
Instead of using the normal superblock, use an alternative superblock specified by
superblock. This option is normally used when the primary superblock has been corrupted.
The location of the backup superblock is dependent on the filesystem's blocksize. For
filesystems with 1k blocksizes, a backup superblock can be found at block 8193; for
filesystems with 2k blocksizes, at block 16384; and for 4k blocksizes, at block 32768.
Additional backup superblocks can be determined by using the mke2fs program using the -n
option to print out where the superblocks were created. The -b option to mke2fs, which
specifies blocksize of the filesystem must be specified in order for the superblock
locations that are printed out to be accurate.
If an alternative superblock is specified and the filesystem is not opened read-only,
e2fsck will make sure that the primary superblock is updated appropriately upon completion
of the filesystem check.
-B blocksize
Normally, e2fsck will search for the superblock at various different block sizes in an
attempt to find the appropriate block size. This search can be fooled in some cases. This
option forces e2fsck to only try locating the superblock at a particular blocksize. If
the superblock is not found, e2fsck will terminate with a fatal error.
-c This option causes e2fsck to use badblocks(8) program to do a read-only scan of the
device in order to find any bad blocks. If any bad blocks are found, they are added to the
bad block inode to prevent them from being allocated to a file or directory. If this
option is specified twice, then the bad block scan will be done using a non-destructive
read-write test.
...[snip]...
-E extended_options
Set e2fsck extended options. Extended options are comma separated, and may take an argument
using the equals ('=') sign. The following options are supported:
...[snip]...
fragcheck
During pass 1, print a detailed report of any discontiguous blocks for files
in the filesystem.
...[snip]...
-n Open the filesystem read-only, and assume an answer of `no' to all questions. Allows
e2fsck to be used non-interactively. This option may not be specified at the same time as
the -p or -y options.
...[snip]...
|
As you can see, I'm pointed for more reading of `man mke2fs', as in the passage, repasting here:
| Code: |
Additional backup superblocks can be determined by using the mke2fs program using the -n
option to print out where the superblocks were created. The -b option to mke2fs, which
specifies blocksize of the filesystem must be specified in order for the superblock
locations that are printed out to be accurate.
|
Excerpts from `man mke2fs':
| Code: |
...[snip]...
OPTIONS
-b block-size
Specify the size of blocks in bytes. Valid block-size values are 1024, 2048 and 4096 bytes
per block. If omitted, block-size is heuristically determined by the filesystem size and the
expected usage of the filesystem (see the -T option). If block-size is preceded by a nega‐
tive sign ('-'), then mke2fs will use heuristics to determine the appropriate block size,
with the constraint that the block size will be at least block-size bytes. This is useful
for certain hardware devices which require that the blocksize be a multiple of 2k.
...[snip]...
-n Causes mke2fs to not actually create a filesystem, but display what it would do if it were to
create a filesystem. This can be used to determine the location of the backup superblocks
for a particular filesystem, so long as the mke2fs parameters that were passed when the
filesystem was originally created are used again. (With the -n option added, of course!)
...[snip]...
|
So the command to try may be:
| Code: |
# mke2fs -t ext4 -n -b /dev/mapper/H_E09
|
The sole parameter that I used when I created the filesystme in question was only `-t ext4'.
| Code: |
uabox ~ # mke2fs -t ext4 -n -b /dev/mapper/H_E09
mke2fs: invalid block size - /dev/mapper/H_E09
uabox ~ #
|
Obviously I didn't read carefully enough.
| Code: |
uabox ~ # mke2fs -t ext4 -n -b -4096 /dev/mapper/H_E09
mke2fs 1.42.12 (29-Aug-2014)
/dev/mapper/H_E09: Operation not permitted while setting up superblock
uabox ~ #
|
However, this doesn't tell me that I'm off my trolley trying to do it. It seems that if it were
read-write, apparently it would acoomplished something.
But it can't yet be read-write, because:
top:
| Code: |
top - 13:07:34 up 1 day, 15:36, 2 users, load average: 4.79, 4.89, 4.85
Tasks: 195 total, 6 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 16.6 us, 13.2 sy, 0.0 ni, 34.7 id, 34.7 wa, 0.1 hi, 0.7 si, 0.0 st
KiB Mem : 16385720 total, 176108 free, 491056 used, 15718556 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15804532 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1917 root 20 0 9100 76 0 R 33.2 0.0 234:24.92 md5sum
1982 root 20 0 9100 76 0 R 32.9 0.0 234:11.17 md5sum
2031 root 20 0 9104 80 0 R 32.6 0.0 234:01.64 md5sum
2030 root 20 0 29908 380 0 D 27.2 0.0 209:10.02 blkls
1981 root 20 0 29908 380 0 R 26.9 0.0 203:55.36 blkls
1916 root 20 0 29908 376 0 R 26.6 0.0 203:08.86 blkls
1148 root 20 0 0 0 0 D 9.0 0.0 120:02.03 usb-storage
...[snip]...
|
The anticipation mixed with uncertainty makes for an unnerving feeling.
The blkls and md5sum jobs are done.
Just why did I get this message, probably at the time those jobs were done:
| Code: |
/bin/ln: failed to create symbolic link
‘/Cmn/autopsy/WCC070-luks-vol/ukrabox/images/sdb1’/bin/ln: failed to create
symbolic link ‘/Cmn/autopsy/WCC070-luks-vol/ukrabox/images/sdb1’: File exists
: File exists
|
I got it on the terminal where Autopsy is running.
And those appear, have a look:
| Code: |
uabox ~ # cat /Cmn/autopsy/WCC070-luks-vol/ukrabox/md5.txt
3B7E4DF6DA0E8BB78283BB66F317689B img1
3B7E4DF6DA0E8BB78283BB66F317689B img2
uabox ~ #
|
to have been two calculations (whch must be down to either Links or Autopsy or something else, in some fashion, way or form, as I didn't start two md5 calculations from the Autopsy browser in Links. The error that I describes still shows in the Autopsy browser in Links, and I couldn't have started that calculation since I started it from that Links window. I have only one other Links window, and in that one I only read the Autopsy help pages from.
But, never mind the extra work that I didn't figure was being done... How do I backup those exact blocks from the entire 3TB partition, just those few, that I obviously need to write over with the superblock that mke2fs seems to be able to recover?
So that if I go wrong, I can retrace my steps without the need to dump from the original HDD all over? To get for myself one huge stage less work, if I mess up next from here?
From the Image Defails selectable from Main manu at the top once I'm in my case, I got this information:
| Code: |
General File System Details
__________________________________________________________________
FILE SYSTEM INFORMATION
File System Type: Ext4
Volume Name:
Volume ID: e4904654fdf535a61e40200e3e125aa3
Last Written at: 2014-12-21 04:46:24 (CET)
Last Checked at: 2013-08-31 16:59:08 (CEST)
Last Mounted at: 2014-12-21 04:43:11 (CET)
Unmounted properly
Last mounted on: /mnt/sdg1
Source OS: Linux
Dynamic Structure
Compat Features: Journal, Ext Attributes, Resize Inode, Dir Index
InCompat Features: Filetype, Extents, Flexible Block Groups,
Read Only Compat Features: Sparse Super, Large File, Huge File, Extra
Inode Size
Journal ID: 00
Journal Inode: 8
__________________________________________________________________
METADATA INFORMATION
Inode Range: 1 - 183148545
Root Directory: 2
Free Inodes: 183148265
Inode Size: 256
__________________________________________________________________
CONTENT INFORMATION
Block Groups Per Flex Group: 16
Block Range: 0 - 732566384
Block Size: 4096
Free Blocks: 513618251
__________________________________________________________________
BLOCK GROUP INFORMATION
Number of Block Groups: 22357
Inodes per group: 8192
Blocks per group: 32768
Group: 0:
Block Group Flags: [INODE_ZEROED, ]
Inode Range: 1 - 8192
Block Range: 0 - 32767
Layout:
Super Block: 0 - 0
Group Descriptor Table: 1 - 175
Group Descriptor Growth Blocks: 176 - 1024
Data bitmap: 1025 - 1025
Inode bitmap: 1041 - 1041
Inode Table: 1057 - 1568
Data Blocks: 9249 - 32767
Free Inodes: 8171 (99%)
Free Blocks: 23517 (71%)
Total Directories: 1
Stored Checksum: 0x1E4B
Group: 1:
Block Group Flags: [INODE_UNINIT, INODE_ZEROED, ]
Inode Range: 8193 - 16384
Block Range: 32768 - 65535
Layout:
Super Block: 32768 - 32768
Group Descriptor Table: 32769 - 32943
Group Descriptor Growth Blocks: 32944 - 33792
Data bitmap: 1026 - 1026
Inode bitmap: 1042 - 1042
Inode Table: 1569 - 2080
Data Blocks: 33793 - 65535
Free Inodes: 8192 (100%)
Free Blocks: 3044 (9%)
Total Directories: 0
Stored Checksum: 0x6C17
Group: 2:
Block Group Flags: [INODE_UNINIT, INODE_ZEROED, ]
Inode Range: 16385 - 24576
...[snip]...
|
but that goes on almost forever, so, to give just the feel of it, these are the last few lines:
| Code: |
Group: 22354:
Block Group Flags: [INODE_UNINIT, BLOCK_UNINIT, INODE_ZEROED, ]
Inode Range: 183123969 - 183132160
Block Range: 732495872 - 732528639
Layout:
Data bitmap: 732430338 - 732430338
Inode bitmap: 732430354 - 732430354
Inode Table: 732431392 - 732431903
Data Blocks: 732495872 - 732528639
Free Inodes: 8192 (100%)
Free Blocks: 32768 (100%)
Total Directories: 0
Stored Checksum: 0xA822
Group: 22355:
Block Group Flags: [INODE_UNINIT, BLOCK_UNINIT, INODE_ZEROED, ]
Inode Range: 183132161 - 183140352
Block Range: 732528640 - 732561407
Layout:
Data bitmap: 732430339 - 732430339
Inode bitmap: 732430355 - 732430355
Inode Table: 732431904 - 732432415
Data Blocks: 732528640 - 732561407
Free Inodes: 8192 (100%)
Free Blocks: 32768 (100%)
Total Directories: 0
Stored Checksum: 0x08F9
Group: 22356:
Block Group Flags: [INODE_UNINIT, INODE_ZEROED, ]
Inode Range: 183140353 - 183148544
Block Range: 732561408 - 732566384
Layout:
Data bitmap: 732430340 - 732430340
Inode bitmap: 732430356 - 732430356
Inode Table: 732432416 - 732432927
Data Blocks: 732561408 - 732566384
Free Inodes: 8192 (100%)
Free Blocks: 4977 (100%)
Total Directories: 0
Stored Checksum: 0x97D3
|
And yet, my zeroed out luks volume, appears to me to be in the 0 group (the first one), and it is logical, because I created the luks-volume files before writing almost anything else on that HDD, in the very first group, so I mostly don't even need but that group and some other for the spare superblock to give to e2fsck, IIUC.
This is a re-paste:
| Code: |
Group: 0:
Block Group Flags: [INODE_ZEROED, ]
Inode Range: 1 - 8192
Block Range: 0 - 32767
Layout:
Super Block: 0 - 0
Group Descriptor Table: 1 - 175
Group Descriptor Growth Blocks: 176 - 1024
Data bitmap: 1025 - 1025
Inode bitmap: 1041 - 1041
Inode Table: 1057 - 1568
Data Blocks: 9249 - 32767
Free Inodes: 8171 (99%)
Free Blocks: 23517 (71%)
Total Directories: 1
Stored Checksum: 0x1E4B
Group: 1:
Block Group Flags: [INODE_UNINIT, INODE_ZEROED, ]
Inode Range: 8193 - 16384
Block Range: 32768 - 65535
Layout:
Super Block: 32768 - 32768
Group Descriptor Table: 32769 - 32943
Group Descriptor Growth Blocks: 32944 - 33792
Data bitmap: 1026 - 1026
|
Notice in group 0 the:
and in group 1 the:
| Code: |
Super Block: 32768 - 32768
|
I think those are the superblocks that e2fsck and mke2fs talk about, and which Neddy mentioned in his kind explanation to me.
Also:
| Code: |
Inode Range: 1 - 8192
Free Inodes: 8171 (99%)
|
which means (because 8192-8171=21) only 21 inodes from this group are used.
| Code: |
Inode Range: 1 - 8192
Block Range: 0 - 32767
Layout:
Super Block: 0 - 0
Group Descriptor Table: 1 - 175
Group Descriptor Growth Blocks: 176 - 1024
Data bitmap: 1025 - 1025
Inode bitmap: 1041 - 1041
Inode Table: 1057 - 1568
Data Blocks: 9249 - 32767
Free Inodes: 8171 (99%)
|
However, since (another short repaste):
| Code: |
CONTENT INFORMATION
Block Groups Per Flex Group: 16
Block Range: 0 - 732566384
Block Size: 4096
|
if the
| Code: |
Data Blocks: 9249 - 32767
|
and that is (32767-9249=23518) 23518*4096=96329728 bytes of data, which in human readable is some 91MB, and so I don't get something here.
What I don't get is, how do I recreate my 400GB (thats 400GB, not just 91MB) luks-volume file? Which exact blocks is that file contained in? It's surely not just this first group, the 0 group! |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 9:54 am Post subject: |
|
|
If I go at this pace of learning things, I reckon, all things being equal, that I might be able to help others in some twenty or thirty years, and not need help. But it's not likely my mind would remain so nimble by that time, since I am almost 60 already. ...
Never mind. Trying to see where I could get some more explanation on the inodes and blocks which keep those exact information that I would overwrite with the actions that I proposed in the latter part of the previous post (such as where fsck couldn't write the superblock on the device in question, because it was readonly), to back those up, so I can revert to this stage (by restoring those back) if I need to start over (saving me some half a day).
The Sleuth Kit Informer
http://www.sleuthkit.org/informer/
never even holds any links at all (other than to CreativeCommons site and to Italian translation).
But the project is far from dead! See it on github:
https://github.com/sleuthkit/
well, more properly:
https://github.com/sleuthkit/sleuthkit
The mailing list also seems to be active (Lord, do I like free software, and how I wish I could contribute at least a little more than just gratefully using the sacred free programs --to me they are kind of sacred because they're so pure for the creators intentions who could have made moneys but care for the common good instead)...
[The mailing list also seems to be active] too:
http://sourceforge.net/p/sleuthkit/mailman/sleuthkit-users/
This outgoing month:
http://sourceforge.net/p/sleuthkit/mailman/sleuthkit-users/?viewmonth=201503
I'll be back to tell if I found more information there or elsewhere to my question above, and in latter part of previous post, which I'm still stuck at. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sun Mar 29, 2015 5:59 pm Post subject: |
|
|
whatever
Last edited by frostschutz on Mon Mar 30, 2015 4:55 pm; edited 1 time in total |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 6:11 pm Post subject: |
|
|
| frostschutz wrote: | It's really hard to help here. I'm all for providing as much information as possible, but you are posting a giant wall of text with very little content. I think I saw three pages worth of text in there about dd being slow, because you did not specify a larger blocksize (obvious to anyone who knows anything about dd and its covered in just about every dd tutorial).
As for your situation, if what you described is correct, then technically your dd command did not overwrite your file; instead it deleted the file and created a new one with the same name. And this might have been stored in a different location. So you might be able to find the original LUKS header and filesystem header in the raw data of your disk. If you are very lucky. But it won't help you much, in the end.
The loss of the file leaves you with a can of worms: You don't know the location of the old file, and since files of this size tend to be fragmented somewhere, you don't know the location of the individual fragments either. But in order to be able to recover anything at all, you must know these locations and offsets precisely.
You are looking at a jigsaw puzzle with possibly thousands of unknown variables. Chance of recovery is very close to zero. |
No, frostschutz, I don't think the action on an existent file deletes that file, no way, you're wrong about that! Almost certain that it does not!
I kept kind of a diary, to not lose exact things that I did. Proves wrong approach for the amount of text, you're right, and sorry for that, but alas it'd cost an non-insignificant overhead on top of the time spent, that I can not afford --working very hard as it is-- and it proves, I think, almost certain as I said, correct way in the semse that you can go back, if you feel like, and find out exact problems, and also figure out that...
(
may point a local link to it, in minutes or longer, but have urgent work first, you will see what urgent work...
)
[and also] you can [figure out that] it's the same file, because the:
superblock would have been written if it wasn't mounted readonly
(still not completely certain, had flops before...) |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 6:22 pm Post subject: |
|
|
**************************************
Just subscribed, so: SOLVED, HELP with subscribing to forum.sleuthkit.org probably NO MORE NEEDED
**************************************
( My subscribing of Sleuthkit Mailing List succeeded; Sleuthkit Forum not yet received mail from. Sleuthkit ML:
Which blocks my very partly zeroed out, recoverable luks volume file occupies?
http://sourceforge.net/p/sleuthkit/mailman/message/33690981/
But I'll leave a shortened post here to witness how regimes and privacy-hostile providers behave.
This is urgent, as I said to frostschutz:
Around 19:30 CET, that's 50 minutes to one hour ago (yet to know to the minute exactly), I completed the form on:
http://forum.sleuthkit.org/
with dillo, successfully.
No mail arrived in my mailbox at HR T-com (HR is the code for Croatia).
What do you think:
a) I filled in wrong data, such as, only possibility, wrong email?
Namely this is what happened:
http://www.CroatiaFidelis.hr/gnu/zerk/Screen_150329_1930_g0n_SleuthkitForums.mkv
[ And this is where I saw the Admin e-mail address:
http://www.CroatiaFidelis.hr/gnu/zerk/Screen_150329_1925_g0n_SleuthkitForums.mkv
]
(all else is excluded, as The Sleuthkit Forum accepted my form; but my email is correct, I checked it over and over)
b) I have something misconfigured in my mail retrieval system (I'm still to post on that, but I can tell you that I get all other mail but not that one activation mail)
c) The regime people sitting at the HR T-com need to see the mail, and approve it and send it or filter it out and not send it to me.
Thank you for your kind attention.
For people reading this to, some time in the future, recover their ovewritten luks device, sorry for this digression, but the only way to fight censorship is by revealing it. Very hard it is, very much work, but if you don't show it, it's darkness, and it's isolation... Thank you for bearing with me!
I've filled this post with links so anyone can see this is my live email address that I filled out in the Sleuthkit form, and I have and may yet do more of it, provide other info that proves what I claim.
And I really love forums, because it's TLS, and I'm free on the forums... Lord!...
EDIT Sun 29 Mar 23:34:33 CEST 2015 START
I have just checked email, confirmation mail has not arrived.
Also retried entering the same values, and it said:
| sleuthkit.org - Registration wrote: |
The username you entered is already in use, please select an alternative. The entered e-mail address is already in use.
Please note that you will need to enter a valid e-mail address before your account is activated. You will receive an e-mail at the address you provide that contains an account activation link.
|
which means, they sent it, and the Regime doesn't let it through. And they probably wouldn't even if I changed the name that I previously entered.
And I have no alternative e-mail, because the miro.rovis@croatiafidelis.hr from the website of my NGO, which I pay the hosting for, they don't let me use, other then very few mails arrive, and only from where I subscribed long ago.
NOTE (much later): This below I leave as a reminder. I might be able to solve my Sleuthkit Forum registration via email, see in top or bottom od this post.
######################## SIMPLE FAVOR PLS #######################
---------------------- %< COPY-PASTE AND SEND --------------------
Please, SleuthKit Forum Admin, Read here:
https://forums.gentoo.org/viewtopic-t-1004014.html#7724054
Miroslav Rovis filled the form, but did not get the mail from you and probably
would not get it, because of censorship from the regime in his country (his
claim).
He kindly ask you to use Gentoo Forums mail to send him his activation email
to Personal Message of Gentoo Forums.
You can send him encrypted: http://www.CroatiaFidelis.hr/4FBAF0AE.asc but
plain text is also fine.
-------------- %< COPY-PASTE AND SEND --------------------
EDIT END
Somebody, anybody who are reading this. You don't have to be a Gentoo Forum member. Anybody! Paste that into an new email, and send it to. probably:
no-reply@sleuthkit.org
( or maybe admin@sleuthkit.org (but I saw, and you can see here, again:
http://www.CroatiaFidelis.hr/gnu/zerk/Screen_150329_1925_g0n_SleuthkitForums.mkv
only no-reply@sleuthkit.org )
:
--------
NOTE (much later): This above I leave as a reminder. I might be able to solve my Sleuthkit Forum registration via email, see in top or bottom od this post.
You want to see more clearly if I'm honest in my claims? Go to:
Postfix smtp/TLS, Bkp/Cloning Mthd, Censorship/Intrusion
https://forums.gentoo.org/viewtopic-t-999436.html#7682770
( My subscribing of Sleuthkit Mailing List succeeded; Sleuthkit Forum not yet received mail from. Sleuthkit ML:
Which blocks my very partly zeroed out, recoverable luks volume file occupies?
http://sourceforge.net/p/sleuthkit/mailman/message/33690981/
But I'll leave a shortened post here to witness how regimes and privacy-hostile providers behave.
**************************************
Just subscribed, so: SOLVED, HELP with subscribing to forum.sleuthkit.org probably NO MORE NEEDED
**************************************
Last edited by miroR on Sat May 09, 2015 5:14 pm; edited 15 times in total |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sun Mar 29, 2015 6:23 pm Post subject: |
|
|
whatever
Last edited by frostschutz on Mon Mar 30, 2015 4:55 pm; edited 1 time in total |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 6:35 pm Post subject: |
|
|
EDIT Mon 4 May 15:27:58 CEST 2015 and Wed 6 May 22:16:55 CEST 2015
What frostschutz had posted in the post above, as I captured in my Screencasts was a longer text... but my attempts seem to have been more effective and correct...
However, you can see what he posted, I have to show it, because otherwise the entire topic now would be b0rked:
http://www.croatiafidelis.hr/foss/cap/cap-150329_GF/Screen_150329_frostschutz.webm
It costs me huge work in comparison to your only inconvenience, gentle reader, of viewing it in reverse, as it was taken by scrolling back (that is part of a 48min screencast. Find it from 0:00:20 to 0:00:30 of this 39s video.
[[ EDIT 2015-05-08 12:08+02:00
Or you can now go to my pastes with my attempt at this frostschutz' tip
]]
RECONSTRUCTION:
| frostschutz wrote: |
To illustrate your issue, the way I understood it, please correct me if I'm wrong.
I created and loop-mounted a 2G ext4 filesystem:
| Code: |
# truncate -s 2G filesystem
# shred -z filesystem
# mke2fs.ext4 filesystem
mke2fs 1.42.12 (29-Aug-2014)
Descarding device blocks: done
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: [...]
Superblock groups stored on blocks:
32768, 98304, 16384, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
# mount -o loop filesystem loop/
|
In this filesystem I created a 1GB LUKS container:
| Code: |
loop # truncate -s 1G container
loop # shred -z container
loop # cryptsetup luksFormat container
WARNING!
========
This will overwrite data on container irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase: foobar
Verify passphrase: foobar
loop # cryptsetup luksHeaderBackup container --header-backup-file container.luks
|
This is to replicate your situation - a LUKS container stored as a file in a filesystem.
Here's what that file's allocation data looks like:
| Code: |
# filefrag -s -v container
Filesystem type is: ef53
File size of container is 1073741824 (262144 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 32767: 34816.. 67583: 32768:
1:
2:
3: [ I'll fill this in, as I learn enough to try these commands ]
4:
5:
6:
7:
8: 237568.. 262143: 296960..
container: 6 extents found
|
Now comes the deadly dd. Since the container is only 1G, I will "overwrite" 128MB. So it's partially overwritten as in your case, as I understood it.
| Code: |
# dd if=/dev/zero of=container bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 0.138092 s, 972 MB/s
|
Here's what the results allocation data looks like:
| Code: |
# filefrag -s -v container
Filesystem type is: ef53
File size of container is 134217728 (32768 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 32767: 321536.. 354303: 32768: last.eof
container: 1 extent found
|
As you can see, it ended up in a different physical location compared to the original container.
The offset of the file produced by dd is 321536-354303. The original offset of the first fragment was 34816-67583.
Conclusion: The file was not overwritten at all, but deleted, and a new file created in its place in a different location. The different location is a matter of luck. The filesystem just picks whatever location is free.
If you wanted dd to overwrite in-place, you'd need the conv=notrunc option.
In the raw data of that filesystem, the original LUKS header can still be found. But what you actually must find in order to be able to restore anything (but the first fragment, if it still exists), is all of the posted earlier:
| Code: |
# filefrag -s -v container
Filesystem type is: ef53
File size of container is 1073741824 (262144 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 32767: 34816.. 67583: 32768:
1:
2:
3: [ I'll fill this in, as I learn enough to try these commands ]
4:
5:
6:
7:
8: 237568.. 262143: 296960..
container: 6 extents found
|
I[t] was just a 1G file but already there are many pieces to the puzzle[. F]or each piece you must know the exact physical offset and length. Without this information the container is just dead and gone.
Hence my suggestion: see if you can find the first piece. The one with the LUKS header, and with any luck if the piece was large enough, with your filesystem metadata. If you have this piece you might have a remote chance at finding some specific other piece. Finding random pieces in general, on the other hand, in nigh impossible.
-----------
To fine pissible LUKS headers on a device:
| Code: |
strings -t d -n 4 /dev/DEVICE | grep -A 3 LUKS
|
In my case the result is:
| Code: |
# strings -t d -n 4 foobar | grep -A 3 LUKS
134746112 LUKS
134746152 xts-plain64
134746184 sha1
134746241 tIMT
--
142606336 LUKS
142606376 xts-plain64
142606408 sha1
142606465 tIMT
|
34816 (the first fragment shown by filefrag above) * 4096 (sector size) = 142606336 so the second hit is the first place we were looking for. And the first hit happens to be the LUKS header backup I created.
|
And below here follows old text, that posted at the original date of the post. I see now that I was wrong.
EDIT END
First thanks for helping me (not only trying to, but helping me, because this is also useful),.
I see your point, but you don't see mine.
Again, I'm not completely sure. but you do need to go through my case in some of the gory details, to understand that the dd job I quickly killed, having noticed within seconds that I was heading for a catastrophy.
And it was 430GB file that luks device/container in that file! Tou don't even delete such a file so quickly on a four year old technology.
Also pls. find these words in one of my previous post in this same topic (from the `man cryptsetup'):
| Code: |
Also note that with a header backup you lose the ability to securely wipe the LUKS device by just overwriting the header and key-slots
|
And I already added my words about how that measns:
that experts can recover content lost like in this case of mine relatively easily and quickly, but I'll be poring over this for so much longer, probably.
And, again, see above, how, also repasting what I wrote:
| Code: |
uabox ~ # mke2fs -t ext4 -n -b -4096 /dev/mapper/H_E09
mke2fs 1.42.12 (29-Aug-2014)
/dev/mapper/H_E09: Operation not permitted while setting up superblock
uabox ~ #
|
As visible, you replied before you saw the entire reply of mine... (Was 1 minute difference, now will be maybe 5, from my last edit of this post, to your post which is next)...
---
The post below, so readers can understand what Neddy Seagoon was replying to in the next post, contained the following text.
EDIT Mon 4 May 14:16:54 CEST 2015
| frostschutz wrote: | Deleting a single file (regardless, of its size) usually takes a fraction of a second. In your case you gave it enough to do that plus write gigabytes of new data.
It's up to you whether to believe me or not, of course...
|
(from my archived screencasts; had to do it, else the whole topic loses a lot in clarity)
EDIT END
Last edited by miroR on Fri May 08, 2015 10:09 am; edited 11 times in total |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sun Mar 29, 2015 6:44 pm Post subject: |
|
|
whatever
Last edited by frostschutz on Mon Mar 30, 2015 4:55 pm; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Sun Mar 29, 2015 6:51 pm Post subject: |
|
|
miroR,
The file is not deleted unless the space it previously occupied has been overwritten.
The pointers to the file have been deleted.
Deteting the pointers is a very fast operation. Its the directory entry, the I-node table and the free space map.
These operations will all be cached, so they will contine after your prompt returns.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 6:59 pm Post subject: |
|
|
| NeddySeagoon wrote: | miroR,
The file is not deleted unless the space it previously occupied has been overwritten.
The pointers to the file have been deleted.
Deteting the pointers is a very fast operation. Its the directory entry, the I-node table and the free space map.
These operations will all be cached, so they will contine after your prompt returns. |
Great to read from you, Senior Gentooer NeddySeagoon
Did you see that I did make progress in understanding you tips?
That was on the bright side. Now on the darker side, I don't understand very clearly what you mean,
Do you mean the pointers are
the directory entry, the I-node table and the free space map ?
And they did get deleted? I know that the data was not, because I showed, in the very first post I think, how the overwritten data was only, let me see:
| me in the first post wrote: |
And of all the 465567744000 of mostly data (plus some overhead that luks needs) 1670819840 in one disk, and 4724903936 in the other will not be recoverable, because they are overwritten... That is some 1.6G on one disk and 4.5G on the other, of the 430G on each respectively, will not be recoverable. The rest should be, somehow, recoverable. Somehow...
|
I've just written a plea on this post:
https://forums.gentoo.org/viewtopic-t-1004014.html#7724054
( if there is no confirmation there that this problem has been solved, kind reader, you can still help. I know how hard it is to get help when you are censored. )
---
EDIT Wed 6 May 22:49:34 CEST 2015: While I'm fine using dillo browser for posting on Gentoo Forums, at the time of my original posting of this, which is when I started using it, and when I felt such calm as dillo is so safe after you used ScmoogleFox --yes Google is sitting in Firefox ans surveilles you-- ...still, when I started using it, I often lost pages and got the previous text in them, ans dillo has a problem with cache. Easily overcome by always refreshing the pages and the forms that you fill in when you open them. Find more at:
Secure: yes, Though Features Missing; and Bugs
http://lists.dillo.org/pipermail/dillo-dev/2015-April/010495.html
So this page I corrected now for the text that I had lost in it.
EDIT END
Last edited by miroR on Wed May 06, 2015 8:56 pm; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Sun Mar 29, 2015 7:29 pm Post subject: |
|
|
miroR,
A filesystem not only stores your data, it stores data about your data too.
This data about your data allows the filesystem to find your data.
When you remove a file, for whatever reason, your data stays on disk until it is overwritten.
Only the data about your data is removed.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sun Mar 29, 2015 9:28 pm Post subject: |
|
|
| NeddySeagoon wrote: | miroR,
A filesystem not only stores your data, it stores data about your data too.
This data about your data allows the filesystem to find your data.
When you remove a file, for whatever reason, your data stays on disk until it is overwritten.
Only the data about your data is removed. |
I know that. I actually managed to plod through, and that was hours of concentrated effort (not too hard to understan, but much harder to memorize to the necessary extent, not like a parrot, but if not the amount of momorizing there, those pieces do not match anymore...)...
[I actually managed to plod through] the:
Ext4 Disk Layout
https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout
There all, well most of, the aswers lie!
But I still think that the inode remained the same. How else on Earth would I get the, and I'm repeating, actually pasting it again, for the third time in this topic, [how else on Earth would I get] the:
| Code: |
uabox ~ # mke2fs -t ext4 -n -b -4096 /dev/mapper/H_E09
mke2fs 1.42.12 (29-Aug-2014)
/dev/mapper/H_E09: Operation not permitted while setting up superblock
uabox ~ #
|
And it is best for the reader, at this point, to go back to where I pasted it for the first time, probably. It would be wrong to repeat all of it here.
I am talking about what I wrote in the post above:
< this same topic >
https://forums.gentoo.org/viewtopic-t-1004014.html#7723732
Pls., somebody; I have checked again, and the activation mail has not arrived; somebody, anybody, do that little favor for me:
https://forums.gentoo.org/viewtopic-t-1004014.html#7724054 |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3131
|
| Posted: Sun Mar 29, 2015 9:44 pm Post subject: |
|
|
Miror, TL;DR
Maximum post size is limited by MySQL database to 64kB, you can stop checking that.
If you want to find help, audience or anything, try to contain your problem within a single screen:
So far I know you have some problem about -possibly-overwritten luks volume, and about dd being slow. And about not being able to express yourself.
No, I'm not going back just to read about dd being slow, even if it makes a cool story.
Now, what the problem actually is? Unless it really is dd being slow, of course. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Sun Mar 29, 2015 10:06 pm Post subject: |
|
|
szatox,
Summary.
Some time ago miroR was creating a new LUKS volume in a file using dd.
Unfortunately, he got the file hame wrong and destroyed an existing LUKS volume in the process.
It was stopped after writing about 90x of 430G. x as I don't recall if it was M or G.
miroR would like to recover the content or the remains oy the original LUKS volume.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sun Mar 29, 2015 11:03 pm Post subject: |
|
|
whatever
Last edited by frostschutz on Mon Mar 30, 2015 4:55 pm; edited 1 time in total |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Mon Mar 30, 2015 3:47 am Post subject: |
|
|
I'm glad some of you are picking up the pieces of what the issue is here, regardless of the fact that the uninviting volume of text I wrote makes you unwilling to read where I, I agree, clumsily exposed my problem.
But I do invite you again, to try and see in the previous text, and it is again at the same link, and it is, in my view, fruit of my correct understanding of what I needed to do...
But I do invite you again, to try and see in the previous text, and it is again at the same link, the paste from my terminal, which I am repasting here:
| Code: |
uabox ~ # cryptsetup --verbose --header /mnt/sdk1/H_E09.bak open /dev/loop0 H_E09
Enter passphrase for /mnt/sdk1/H_E09.vol:
Key slot 0 unlocked.
Command successful.
uabox ~ #
|
It's a lot to re-read for me to, but why don't you notice that it's mostly quotes, and the quotes relevant for my particular recovery needs here, from the respective manuals that deal with these issues: `man cryptsetup', `man mke2fs'... and only then, I follow those manual exceprts with my musings...
And those musings of mine brought me to, IIUC (I've had flops, so I'm not 100% certain in alnost nothing anymore)...
And those musings of mine brought me to, IIUC:
opening of my luks volume in question?
Can't you see that I did open my luks volume in question in that long post with so volumionuos quotes from `man cryptsetup' and `man mke2fs' followed by my tentatinve understanding of those exceprts, the understanding that appears to have proved to have been true, because I opened that luks volume?
And, after I opened the luks volume, I probably (but I had a few flops, don't want to make claims of certainty) would have ben able to restore the superblock with that command that I already repeated three times, and that should best be read in the link above in this post...
And, after I opened the luks volume, I probably would have ben able to restore the superblock on that open luks volume filesystem, had it not been readonly mounted.
And this brings up to the snag, the obstacle, which I'm stuck at. and which is exactly about:
| frostschutz wrote: |
...[snip]...
With luck, your original LUKS header and filesystem superblock might still be intact. Unfortunately you still need the whole picture (whole list of extents of your file) in order to be able to recover but... it might be a start.
|
Yes, it is, not intact on the disk, but, with the `--header' option to the cryptsetup command, applicable, my original LUKS header.
And so the filesystem superblock appears, not intact, but recoverable, as I wrote above.
But, you are right, I still need the whole picture (whole list of extents of your file) in order to be able to recover...
And the exact snag is (but I may also go without it, and lose that half a day that I would be able to save if I were to accomplish the following):
I would like to be able to back up only those extents in my file (I'm able to use seek and skip options of dd), only those that would be overwritten once I mouint the entire 3TB partition readwrite, and which would be overwritten with that command, for the fourth time (for clarity, not for any other reason):
| Code: | uabox ~ # mke2fs -t ext4 -n -b -4096 /dev/mapper/H_E09
mke2fs 1.42.12 (29-Aug-2014)
/dev/mapper/H_E09: Operation not permitted while setting up superblock
uabox ~ # |
(Only, sure, that command would really restore the cuperblock, once I had previous to running it, mounted the entire 3TB partition which hold that luks-volume file, among other data, read-write and not read-only as it currently is.)
My snag, my obstacle to maybe solve, is my inability to learn somehow the exact sectors, or blocks, or extents, to dd dump with `dd seek=N ...' to be able to, if my attempt is not successful, recreate this same partition verifiable to have the MD% as can be seen above:
| Code: | | 3B7E4DF6DA0E8BB78283BB66F317689B img1 |
But I may forgo that goal... (I have explained now maybe in cleared terms what I tried to previously explain, that snag, that obstacle, in vaguer terms. for consistency's sake, if you will.)
Now, not many people are reading this, because this is already far too complex for many of even Gentoo users to easily understand, but I plea for help again: pls. help me out of my provider's concrete case of censorship here; thank you!
EDIT Mon 4 May 17:23:19 CEST 2015:
---
Do you see the post below? While frostschutz was simply not right about dd having deleted the file in a previous post (tired really from this reconstruction, if you read this to solve a problem of yours you will have noticed it...)...
While he was completely wrong there, and that made me suspect of his expertize, in this text right below here, he proposed the solution that also Atila from Sleuthkit ML said was right.
So I have to quote the text below (currently it is on the first page of this topic, if after this posting it shows on the second, I'll repost it there too) completely out of order, and the fault is not mine.
| frostschutz wrote: |
hexdump -C H_E09.vol
|
I hope his help will still remain vaulable. I searched for this, and why on Earth make me spend so much more time by deleting posts?
EDIT END
Last edited by miroR on Mon May 04, 2015 3:23 pm; edited 1 time in total |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Mon Mar 30, 2015 9:45 am Post subject: |
|
|
whatever
Last edited by frostschutz on Mon Mar 30, 2015 4:55 pm; edited 1 time in total |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
