Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO encrypt an existing filesystem with dmcrypt and dd
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Thu Feb 08, 2007 9:35 pm    Post subject: HOWTO encrypt an existing filesystem with dmcrypt and dd Reply with quote

I've been meaning to post this for a while.

Messing around with dd and dmcrypt, I discovered it was almost pathetically easy to encrypt an existing filesystem/partition.
I'm sure this is nothing new and quite obvious to some, but I've never seen it mentioned anywhere, so I thought in was worthy of a little howto.

    If anything goes wrong, you could easily lose everything on the filesystem. This should go without saying, and cannot be stressed enough.
    I've tested this with small filesystems I created just for this, and used it to convert a large existing filesystem (which I still have encrypted), and have encountered no problems.
    I do believe this is a lot safer than trying to convert an existing filesystem to a different type (eg ext3 -> reiser4),
    however you would be a fool to try this without backing up anything important to another device first.
    If anything goes wrong, you'll know prettty much immediately. Once it has been successfully converted, then nothing else can really go wrong (apart from forgetting the keyphrase :wink: ).

I am still a novice at, well, everything, so if any more (or even less) experienced users have any reasons why this is a really bad idea and should not be attempted under any circumstances, please share them.

Prerequisites;
You'll need dmcrypt enabled in the kernel, as well as the ciphers you wish to use (aes, blowfish, etc).
In menuconfig, dmcrypt is under "Device Drivers --> Multi-device support (RAID and LVM) --> Crypt target support", you'll need Device Mapper support to make it available.
Under "Cryptographic options" select the ciphers you want to play with. "CBC support" should be selected for you, and you'll also need the SHA256 digest algorithm if you want essiv.
Apart from that the only thing you need is cryptsetup or cryptsetup-luks, both are in portage.

Okay, now that I've scared off just about anyone who tried to read this (through boredom mostly), lets get to the good stuff;

Replace ${INDEV} with the device you want to encrypt (eg /dev/sda4).
If you're lazy, you can just enter "INDEV=/dev/sda4" in a root shell, and then copy and paste the rest.
Also set DMDEV equal to the name of the dmcrypt device mapping (eg "INDEV=sda4_crypt").

    First of all, shouldn't need to be said, but make absolutely sure the filesystem is not mounted;
    Code:
    umount ${INDEV}

    For piece of mind, I strongly suggest making an md5sum of the partition to be encrypted;
    Code:
    INDEV_MD5=`md5sum ${INDEV}`

    Next, create the dmsetup mapping, using whatever cryptsetup options you prefer;
    Code:
    cryptsetup -y -v -c aes-cbc-essiv:sha256 -s 256 create  ${DMDEV} ${INDEV}

    Now copy from the plain partition to the dmcrypt mapping (of the same device) with dd (this is were the magick happens);
    Code:
    dd if=${INDEV} of=/dev/mapper/${DMDEV} bs=512 conv=notrunc

    Once that has completed, check the md5sum of the new mapping against the one you made earlier;
    Code:
    md5sum /dev/mapper/${DMDEV}
    echo ${INDEV_MD5}

    If these match, then everything went smoothly.
    At this stage I'd recommend removing the cryptsetup mapping and setting it up again (to make sure you remember the right keyphrase), running fsck, and finally mounting your newly encrypted filesystem.
    Code:
    cryptsetup remove ${DMDEV}
    cryptsetup -v -c aes-cbc-essiv:sha256 -s 256 create  ${DMDEV} ${INDEV}
    fsck ${INDEV}

    You can also compare the md5sum of the raw device (md5sum ${INDEV}) if you want, just to prove to yourself that it has changed.



The most obvious use of this is to encrypt previously unencrypted partitions/filesystems, however for those already using dmcrypt I can think of a couple of more uses;

It could be used to encrypt a already encrypted filesystem, but allowing you to change either the keyphrase (in case you decided it was too weak, or may have been compromised) or the cipher (changing from blowfish to aes or from cbc-plain to cbc-essiv, for example);
Code:
cryptsetup -v -c aes-cbc-plain -s 128 create ${DMDEV} ${INDEV}
cryptsetup -y -v -c aes-cbc-essiv:sha256 -s 256 create ${DMDEV}-2 ${INDEV}
dd if=/dev/mapper/${DMDEV} of=/dev/mapper/${DMDEV}-2  bs=512 conv=notrunc

It could also be used to permenantly decrypt a filesystem, if you believed it was no longer required or too much trouble,
basically the reverse of the earlier procedure;
Code:
dd if=/dev/mapper/{DMDEV} of=${INDEV} bs=512 conv=notrunc


I've tested both of these out as well and they worked perfectly.

The main disadvantage of this is that it cannot be used with the luks extensions (I don't use them anyway) as luks creates a header at the start of the partition, so dd would be overwritting blocks which were not read yet. You wont be able to use the cryptsetup offset option for the same reason.
I can think of a couple of possible ways around this, but they would entail a lot of work and a far higher risk of losing everything.

It might be possible to use this to convert to and from loop-aes and cryptoloop encrypted partitions,
but I haven't done any testing on this.

It could of course be used to convert the root partition, but will need work in order to get it to boot, and this has been covered in length elsewhere.
Take a look at gentoo-wiki.com/Index:Security for details (scroll down to the filesystem section).
There's also a lot of useful related posts in the forums, particularly under Documentation, Tips & Tricks.

So, anyone interested in this?
Comments, questions, insults?

The fine print;
Hopeless shall in no event be liable for any claims, charges, demands, damages, liabilities, losses, and expenses of whatever nature and howsoever arising, including, but not limited to,
any damages, loss of use, loss of data, loss of income or profit, loss of or damage to property, claims of third parties, or other losses of any kind or character arising out of or in connection with the use of this HOWTO.
Users acknowledge that the risk of injury from the foregoing rest entirely with them. Moreover, users assume total responsibility for establishing procedures for data back up as you consider necessary.

This does not affect your statutory rights.

_________________
"You have to invite me in"
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Sat Jun 02, 2007 8:27 am    Post subject: Reply with quote

maybe just to note that this technique WILL NOT WORK with cryptsetup-luks; since the LUKS-part will overwrite the first few sectors of your disk.
This also means that your cryptsetup-luks-partition will be SMALLER than the original.
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Thu Jun 07, 2007 12:35 am    Post subject: Reply with quote

nielchiano wrote:
maybe just to note that this technique WILL NOT WORK with cryptsetup-luks; since the LUKS-part will overwrite the first few sectors of your disk.
This also means that your cryptsetup-luks-partition will be SMALLER than the original.

I already pointed that out;
Hopeless wrote:
The main disadvantage of this is that it cannot be used with the luks extensions (I don't use them anyway) as luks creates a header at the start of the partition, so dd would be overwritting blocks which were not read yet. You wont be able to use the cryptsetup offset option for the same reason.
I can think of a couple of possible ways around this, but they would entail a lot of work and a far higher risk of losing everything.
Although it is definitely worth emphasizing.

Anyway, you can use it with cryptsetup-luks, you just can't use it to convert to a luks format partition.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Thu Jun 07, 2007 8:27 am    Post subject: Reply with quote

Hopeless wrote:
I already pointed that out;

sorry :oops: should have read your post better
Back to top
View user's profile Send private message
c4
Guru
Guru


Joined: 21 Feb 2004
Posts: 312
Location: Sweden

PostPosted: Mon Jul 16, 2007 8:13 pm    Post subject: Reply with quote

Thanks for a nice howto!
I've been thinking about a setup similar to this and your guide was just the inspiration I needed.
Got this running on a spare partition and it worked just fine.
_________________
AMD64 Gentoo Hardened server
AMD64 Xubuntu Desktop
X86 Dreamlinux Vaio laptop
Back to top
View user's profile Send private message
jkcunningham
l33t
l33t


Joined: 28 Apr 2003
Posts: 649
Location: 47.49N 121.79W

PostPosted: Thu Aug 09, 2007 1:53 am    Post subject: Reply with quote

Nice howto.

Just out of curiosity, why is the dd step so slow? I'm in the middle of converting a 384G hard drive (day 2) and its going to take another 52 hours to finish. Its averaging 1.5 Mbps. But I'm converting a USB 2.0 hard drive partition, and I checked the rate on that previously - its registering at 480 Mbps according to usbview.

And if there's a power glitch (out here in the woods - not uncommon) I'm back to start all over again. Surely there's a faster way to do this? This was a blank disk to begin with.

--Jeff
Back to top
View user's profile Send private message
c4
Guru
Guru


Joined: 21 Feb 2004
Posts: 312
Location: Sweden

PostPosted: Thu Aug 09, 2007 7:03 pm    Post subject: Reply with quote

Took me something like 4 whole days to convert a 2TB raid partition, but it only contained 1.5 TB of old backups.
The box was a amd64-3800 with 1gb ram. I figured that the converting time had to do with both the reading&writing from the drive and then the additional time it took for the system to encrypt the files.

I have not tried to encrypt anything else besides this raid partition, but I would imagine that running such a task on a slower cpu would take more time to perform the encryption.
_________________
AMD64 Gentoo Hardened server
AMD64 Xubuntu Desktop
X86 Dreamlinux Vaio laptop
Back to top
View user's profile Send private message
jkcunningham
l33t
l33t


Joined: 28 Apr 2003
Posts: 649
Location: 47.49N 121.79W

PostPosted: Fri Aug 10, 2007 2:19 am    Post subject: Reply with quote

Well I'm apparently encrypting an empty half-terabyte disk. I guess the reason it is necessary is to end up with an encrypted superblock, but it seems like there ought to be a way to encrypt that by itself in the case of a new partition. This process doesn't lend itself to doing it very often. :o
Back to top
View user's profile Send private message
fangorn
Veteran
Veteran


Joined: 31 Jul 2004
Posts: 1886

PostPosted: Fri Aug 10, 2007 5:48 am    Post subject: Reply with quote

This is a nice idea, but it has some downsides.

1. cryptsetup without the LUKS extensions is known to be weaker to cryptographic attacks.
2. for half empty partitions, you have only half the partition filled up with random data.

Nevertheless, I will keep that in mind when having to do the next reconfiguration of discmanager.
_________________
Video Encoding scripts collection | Project page
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Tue Aug 14, 2007 3:34 pm    Post subject: Reply with quote

jkcunningham wrote:
Just out of curiosity, why is the dd step so slow? I'm in the middle of converting a 384G hard drive (day 2) and its going to take another 52 hours to finish. Its averaging 1.5 Mbps. But I'm converting a USB 2.0 hard drive partition, and I checked the rate on that previously - its registering at 480 Mbps according to usbview.
Encryption is a very cpu-intensive task, and with this howto you are reading from then and writing to the exact same area of the disk, which is bound to have a negative effect on io performance.

Having said that, I have a feeling using a blocksize of 4096 rather than 512 bytes would speed the process up some, more on that in a bit.
jkcunningham wrote:
And if there's a power glitch (out here in the woods - not uncommon) I'm back to start all over again. Surely there's a faster way to do this? This was a blank disk to begin with.
I actually think it's possible to recover from a power output during this, but it would require a fair bit of work (mostly trying to figure out at what exact point in the conversion the power failed) but fortunately it's not something I've needed to put to the test (yet).

Anyway, if this was blank disk to begin with, then using this howto doesn't really make sense, as it would be easier to simply create a new dmcrypt mapping, and then create a brand new filesystem on that.
This would actually take almost no time at all, however it is typically recommended to overwrite the entire partition with random data first, which would probably take just as long.

Even the fastest of todays cpus take ages to generate gigabytes of "random" data via /dev/urandom.

fangorn wrote:
This is a nice idea, but it has some downsides.

1. cryptsetup without the LUKS extensions is known to be weaker to cryptographic attacks.
2. for half empty partitions, you have only half the partition filled up with random data.

Nevertheless, I will keep that in mind when having to do the next reconfiguration of discmanager.
Point number 2 is quite valid, although what you will actually get is the free space on the partition/filesystem (presumably zero's, but probably including "deleted" file data too) will be encrypted too.
My point being, you should not find a bunch of zeros or unencrypted data on the underlying partition.
However, it's not as secure as having the empty space filled with random data prior to encryption, so I think I need to add something to the howto to address that.

How about using `dd if=/dev/urandom of=somefile` or simply `cat /dev/urandom > somefile`, where somefile is an arbitrary file on the filesystem you wish to convert?

The file would just increase in size until using up all the free space on the drive, filing it with random data, which would become encrypted during the conversion.
It wouldn't actually matter if this was done before or after the conversion.
Of course, this is grossly inefficient, but I can't really see any other way of doing this (and it is pretty much synonymous with filling a new partition with random data before creating an encrypted filesystem on top of it).

Anyway, Fangorn about you're first point, how is cryptsetup without LUKS actually weaker?
I've heard the same elsewhere, but haven't seen anything to back it up.

The only real difference is that the actual key used for the encryption with luks is created from random data, rather than a hashed keyphrase with plain cryptsetup, so it depends on just how secure the keyphrase is (and actually there is nothing stopping you from using random data for the keyphrase with plain cryptsetup, only it'd be a bitch to remember :twisted: ).

For example, create both two encrypted loopback files, one with luks extensions, the other without.
Now run `dmsetup table`, and you'll get the actual keys used to encrypt the data, along with all other settings.

It fact, you should be able to use this to set up a plain cryptsetup mapping and bypass the luks stuff altogether.
Admittedly, I haven't tested this (trying to use the luks extensions causes cryptetups to segfault on my amd64 box), so I could be talking through my arse.


Anyway, the original reason I'm replying to this is that I just stumbled across this one the "official" cryptsetup-luks wiki, which basically outlines the same procedure as my howto, only it was written in 2005. :?

In that guide it suggests using a blocksize of 4k, so presumably there are no problems with it.
I just tested with a 1.2 GB loopback file, I got about 14 MB/s with bs=512 and 17 MB/s with 4k, and just for reference I got over 50 MB/s when copying from /dev/zero rather than the unencrypted underlying partition, which implies the problem is with the IO, and the encryption has a neglible (if any) effect.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Tue Aug 14, 2007 4:08 pm    Post subject: Reply with quote

Hopeless wrote:
It fact, you should be able to use this to set up a plain cryptsetup mapping and bypass the luks stuff altogether.
You'll also need to skip a few blocks (definitely more than one, but I don't know how much) in the beginning of the disk, the place where LUKS does its thing.
(I haven't tested it either)
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Tue Aug 14, 2007 4:43 pm    Post subject: Reply with quote

nielchiano wrote:
Hopeless wrote:
It fact, you should be able to use this to set up a plain cryptsetup mapping and bypass the luks stuff altogether.
You'll also need to skip a few blocks (definitely more than one, but I don't know how much) in the beginning of the disk, the place where LUKS does its thing.
(I haven't tested it either)

I think you took this a little out of context, what I was trying to say was the `dmsetup table` output (not anything in my howto) was all you need to bypass the luks stuff, and the offsets required to skip the luks header should be included in that data.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Wed Nov 21, 2007 3:10 pm    Post subject: Reply with quote

Just a bump to tell you that I've just used this to convert 300 GB filesystem, changing from aes-cbc-essiv:sha256 to aes-lrw-benbi:sha256, and changing the key while I was at it.

It worked perfectly.

I only used a bs value of 512, and it took about 6 hours, with an average rate of 15 MB per second;
Code:
 # time dd if=/dev/mapper/bleh1 of=/dev/mapper/bleh2 bs=512
629137527+0 records in
629137527+0 records out
322118413824 bytes (322 GB) copied, 21235.1 s, 15.2 MB/s

real    353m55.409s
user    1m52.969s
sys     62m6.107s


edit: Just thought of another use for this, I'm about to leave for a while and I giving my hardware to a friend to play around with (with a few empty partitions to install on), and I'm going to use this to encrypt every unecrypted partition/filesystem on the hard drives.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Wed Jan 16, 2008 6:51 am    Post subject: Reply with quote

I might have read 'man dd' wrong, but it says:
Quote:
[...]
oseek=n Seek on the output file n blocks. This is synonymous with
seek=n.

seek=n Seek n blocks from the beginning of the output before copying.
On non-tape devices, an lseek(2) operation is used. Otherwise,
existing blocks are read and the data discarded. If the user
does not have read permission for the tape, it is positioned
using the tape ioctl(2) function calls. If the seek operation
is past the end of file, space from the current end of file to
the specified offset is filled with blocks of NUL bytes.

skip=n Skip n blocks from the beginning of the input before copying.
On input which supports seeks, an lseek(2) operation is used.
Otherwise, input data is read and discarded. For pipes, the
correct number of bytes is read. For all other devices, the
correct number of blocks is read without distinguishing between
a partial or complete block being read.
[...]


AFAIK, the LUKS-header has a fixed size, so it could be possible to just skip them in the output-dev?

Wrong?
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Wed Jan 16, 2008 7:20 am    Post subject: Reply with quote

ph030 wrote:
I might have read 'man dd' wrong, but it says:
Quote:
[...]
oseek=n Seek on the output file n blocks. This is synonymous with
seek=n.

seek=n Seek n blocks from the beginning of the output before copying.
On non-tape devices, an lseek(2) operation is used. Otherwise,
existing blocks are read and the data discarded. If the user
does not have read permission for the tape, it is positioned
using the tape ioctl(2) function calls. If the seek operation
is past the end of file, space from the current end of file to
the specified offset is filled with blocks of NUL bytes.

skip=n Skip n blocks from the beginning of the input before copying.
On input which supports seeks, an lseek(2) operation is used.
Otherwise, input data is read and discarded. For pipes, the
correct number of bytes is read. For all other devices, the
correct number of blocks is read without distinguishing between
a partial or complete block being read.
[...]


AFAIK, the LUKS-header has a fixed size, so it could be possible to just skip them in the output-dev?

Wrong?

Yes, you could do that, but then you would be overwriting blocks which have not yet been read.

What you would need to do is create some kind of "buffer", where you copy the next piece of data before overwriting it with the "previous" part.

I'm sorry that's not too clear, both I hope you get what I mean.

I don't think it's possible with dd itself, but it would be doable with a fairly simple shell script which repeated calls dd.

The real problem here though is the fact that with luks the actual key used for encryption is actually stored in the header and cannot be changed, therefore this would only be useful for "removing" the encryption altogether...

I'm not too experienced with luks though, so I could be wrong here.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Thu Jan 17, 2008 6:14 am    Post subject: Reply with quote

Quote:
The real problem here though is the fact that with luks the actual key used for encryption is actually stored in the header and cannot be changed, therefore this would only be useful for "removing" the encryption altogether...
What do you mean by that? It's no problem to change/add/remove a passphrase without reencrypting or did I understand you wrong(again ;) )?

Anotherthing, what is the purpose of using dd here? I mean, I could create the fs myself and then use 'tar -cfp ...' or similar things, which should definetly be faster than dd.
Back to top
View user's profile Send private message
h0mer`-
Apprentice
Apprentice


Joined: 02 Aug 2004
Posts: 215

PostPosted: Thu Jan 17, 2008 11:29 am    Post subject: Reply with quote

just to clarify... is it possible to encrypt a root partition with the method described here?
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Thu Jan 17, 2008 3:53 pm    Post subject: Reply with quote

h0mer`- wrote:
just to clarify... is it possible to encrypt a root partition with the method described here?

Sure, just make sure that it is not the CURRENT root partition (since that would be in use). But you can boot from LiveCD or another partition and do this trick
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Thu Jan 17, 2008 4:38 pm    Post subject: Reply with quote

[quote="Hopeless"]
ph030 wrote:

Yes, you could do that, but then you would be overwriting blocks which have not yet been read.

What you would need to do is create some kind of "buffer", where you copy the next piece of data before overwriting it with the "previous" part.

I'm sorry that's not too clear, both I hope you get what I mean.

I don't think it's possible with dd itself, but it would be doable with a fairly simple shell script which repeated calls dd.

The real problem here though is the fact that with luks the actual key used for encryption is actually stored in the header and cannot be changed, therefore this would only be useful for "removing" the encryption altogether...

I'm not too experienced with luks though, so I could be wrong here.


In the bad old days of DOS, the real "OS" was in 2 files, 1 of which had to have at least its first portion be in specific consecutive sectors on the disk. Specifically, the Master Boot Record could only start loading sectors at a fixed offset from the start of the partition. The first portion of DOS was in those sectors, and by the time those first few sectors were loaded, it had a mini-filesystem that could locate the rest of the first file for loading, as well as load the second file.

The essence of this is that there was at least 2 programs (format and sys) that knew and manipulated true physical locations on the disk, and how to bridge the physical sector / filesystem gap. Obviously defrag would be a third such program.

In Linux there is are least resize2fs, mkfs, and tune2fs that have this same type of knowledge.

Seems to me that it should be possible to write a program that could put a blank file or space at the start of a partition that happens to be the right size for the LUKS header. Any file that happened to be there could just be copied, and it would go elsewhere on the disk. IMHO the hard part would then be getting the LUKS header into that space as you encrypt in-place. This would have to be a root operation, but then the rest of it is, anyway.
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Thu Jan 17, 2008 8:06 pm    Post subject: Reply with quote

ph030 wrote:
Quote:
The real problem here though is the fact that with luks the actual key used for encryption is actually stored in the header and cannot be changed, therefore this would only be useful for "removing" the encryption altogether...
What do you mean by that? It's no problem to change/add/remove a passphrase without reencrypting or did I understand you wrong(again ;) )?

How luks works is that the actual key used for encrypting/decrypting the data is stored in the luks header, where it is encrypted itself.

When you enter your passphrase on luks, it is used to decrypt the master key, which is then used to decrypt the actual data.

This is why you can have multiple passphrases and change your passphrase in luks, as it doesn't involve changing the master key (just storing another encrypted copy of it).

It also means this method can't be used to change the encryption key with luks, although it's not really as necessary as it is with plain cryptsetup.

How the plain cryptsetup works is a hash of the passphrase you entered is directly used as the actual key to encrypt/decrypt the data.

ph030 wrote:
Anotherthing, what is the purpose of using dd here? I mean, I could create the fs myself and then use 'tar -cfp ...' or similar things, which should definetly be faster than dd.

The purpose here is that your method would require another disk/partition to create the tarball on while you recreate the filesystem atop dmcrypt on the original partition, whereas this method does not.

Your method is basically the more conventional (and safer) way of doing the exact same thing.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Thu Jan 17, 2008 8:16 pm    Post subject: Reply with quote

depontius wrote:
Hopeless wrote:
Yes, you could do that, but then you would be overwriting blocks which have not yet been read.

What you would need to do is create some kind of "buffer", where you copy the next piece of data before overwriting it with the "previous" part.

I'm sorry that's not too clear, both I hope you get what I mean.

I don't think it's possible with dd itself, but it would be doable with a fairly simple shell script which repeated calls dd.

The real problem here though is the fact that with luks the actual key used for encryption is actually stored in the header and cannot be changed, therefore this would only be useful for "removing" the encryption altogether...

I'm not too experienced with luks though, so I could be wrong here.


In the bad old days of DOS, the real "OS" was in 2 files, 1 of which had to have at least its first portion be in specific consecutive sectors on the disk. Specifically, the Master Boot Record could only start loading sectors at a fixed offset from the start of the partition. The first portion of DOS was in those sectors, and by the time those first few sectors were loaded, it had a mini-filesystem that could locate the rest of the first file for loading, as well as load the second file.

The essence of this is that there was at least 2 programs (format and sys) that knew and manipulated true physical locations on the disk, and how to bridge the physical sector / filesystem gap. Obviously defrag would be a third such program.

In Linux there is are least resize2fs, mkfs, and tune2fs that have this same type of knowledge.

Seems to me that it should be possible to write a program that could put a blank file or space at the start of a partition that happens to be the right size for the LUKS header. Any file that happened to be there could just be copied, and it would go elsewhere on the disk. IMHO the hard part would then be getting the LUKS header into that space as you encrypt in-place. This would have to be a root operation, but then the rest of it is, anyway.

I don't think it's quite that simple, I'm fairly sure the very beginning (ie the first few sectors, at least) of any linux filesystem contains the basic information of (and needed by) the filesystem itself, which not something which could or should be moved...

In other words, the area we would need to move would be used by the filesystem itself, and not one of the files on the filesystem.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3509

PostPosted: Thu Jan 17, 2008 8:51 pm    Post subject: Reply with quote

Hopeless wrote:

I don't think it's quite that simple, I'm fairly sure the very beginning (ie the first few sectors, at least) of any linux filesystem contains the basic information of (and needed by) the filesystem itself, which not something which could or should be moved...

In other words, the area we would need to move would be used by the filesystem itself, and not one of the files on the filesystem.


I think I need to understand a little more about LUKS. Since you say "LUKS puts its information at the beginning of the..." I would tend to think that means either the beginning of the filesystem, or the beginning of the partition. If the former, then LUKS has just put its information in a specific set of blocks, which is essentially what I have suggested. If the latter, then LUKS is essentially a "container partition", meaning it sits *raw* inside a partition, and then itself acts like a partition to the filesystem it's hosting.

But it looks to me as if LUKS is really just key management added to cryptsetup, which is in turn based on dm-crypt and dmsetup. Certainly dm-crypt would be in the container business, but the man page of dmsetup (which cryptsetup/cryptsetup-LUKS simply wraps around, according to the Wiki.) doesn't appear to have any sort of knobs that would let you tune such a thing. That gives me the immediate impression that LUKS is stuffing its information in some sort of file pinned near the beginning of the filesystem, about as soon after the superblock stuff as can be done. Since ext2/3 don't simply start growing from the bottom, like some other filesystems, it's probably possible to get quite close to the beginning, right after the superblock. I don't know where ext3 keeps the journal, though I know it pulls tricks similar to what I'm suggesting. (If you start with ext2, use "tune2fs -j" to add a journal, you'll have a ".journal" file visible in the root of that filesystem, until you reboot. After a reboot the journal will still be there, but completely hidden.) I would also suspect that the ext3 journal is NOT at the beginning of the filesystem, but either distributed or at the center, to optimize seek performance.
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Thu Jan 17, 2008 9:34 pm    Post subject: Reply with quote

I'm no LUKS expert either, but according to the info I read, LUKS is acting more like a "header" It occupies the first N blocks of the partition. The crypto-data starts at sector N+1.
You can check that by creating a non-LUKS cryptomap and compare the offsets in dmtable:
Code:
loopcrypto: 0 20480 crypt aes-cbc-plain XXXXXXXX 0 7:0 0

Code:
loopcryptoLUKS: 0 19448 crypt aes-cbc-essiv:sha256 XXXXXXXX 0 7:0 1032

(format is documented here: http://www.saout.de/misc/dm-crypt/)
You can see that the luks-partition is offset by 1032 sectors from the beginning. In other words: the first 1032 sectors contain the LUKS header specifying the different keys that can be used to unlock the partition.
The (encrypted) filesystem doesn't even SEE the LUKS-data, just like it doesn't see the previous partition of the disk.
so this is correct:
depontius wrote:
then LUKS is essentially a "container partition", meaning it sits *raw* inside a partition, and then itself acts like a partition to the filesystem it's hosting
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Thu Jan 17, 2008 11:14 pm    Post subject: Reply with quote

Quote:
The purpose here is that your method would require another disk/partition to create the tarball on while you recreate the filesystem atop dmcrypt on the original partition, whereas this method does not.
No, that's definetly not necessary, you can just pipe it, like this
Code:
tar -c src/ | tar -C dest/ -xv
like described in http://gentoo-wiki.com/TIP_Fast_Copy - so there needs to be "-p" to keep permissions, but that should be it.
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Fri Jan 18, 2008 1:02 am    Post subject: Reply with quote

ph030 wrote:
Quote:
The purpose here is that your method would require another disk/partition to create the tarball on while you recreate the filesystem atop dmcrypt on the original partition, whereas this method does not.
No, that's definetly not necessary, you can just pipe it, like this
Code:
tar -c src/ | tar -C dest/ -xv
like described in http://gentoo-wiki.com/TIP_Fast_Copy - so there needs to be "-p" to keep permissions, but that should be it.

Ah, I get what you're saying now, but it wouldn't work.

The problem is that the untar'ed files could be written anywhere on the filesystem/disk, overwriting other files which may already have been copied, or have yet to be read by the first tar.

That is why dd is necessary, because it copies to the exact same position on the disk, therefore it only overwrites what it has just read and copied.

Make sense?
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum