| View previous topic :: View next topic |
| Author |
Message |
midnite
Guru
Joined: 09 Apr 2006
Posts: 435
Location: Hong Kong
|
 Posted: Wed Mar 16, 2022 2:20 pm Post subject: ZFS on LUKS on MDADM |
 |
|
I have been planning this for quite a while, until now I have a chance to do it. I am about to setup a system with ZFS, LUKS with detached header (also guides in archlinux and other) which has deniable encryption), and MDADM RAID 1 mirroring.
With reference to this post in serverfault, I am about to setup a similar system. In fact the answer suggests the "comparing to" design.
| Target design: | ZFS <-top
|
LUKS (whole "disk", detached header)
|
mdadm (RAID 1 mirror)
|
Raw disk x2 <- bottom |
| Comparing to: | ZFS mirroring <-top
|
LUKS (whole disks, two disks individually, detached headers)
|
Raw disk x2 <- bottom |
To my understanding, the upper target design has the benefit of encrypt once by LUKS and keep one key, while comparing to the lower design LUKS has to encrypt twice and keeping two different keys.
Q) In the lower design, two disks are encrypted separately by LUKS. Is it possible to use the same key (thus keeping only one header) for two disks? (I think theoretically possible, but no solution on Internet yet.) If yes, can we tell LUKS to encrypt only once so it saves redundant processing times? (I think it is not possible.)
Q) In the upper target design, ZFS has checksums so when there is a bit error, ZFS will know it. But can ZFS recover from bit errors? I think not. In the view of ZFS, there is no redundant data. ZFS can only report there is an error, then let MDADM to recover it.
Any suggestions to the designs?
Thanks a lot in advance.
_________________
- midnite. |
|
| Back to top |
|
 |
midnite
Guru
Joined: 09 Apr 2006
Posts: 435
Location: Hong Kong
|
| Posted: Mon May 16, 2022 11:14 pm Post subject: |
|
|
Lemme add some quick updates on this project.
ZFS mirror or ZFS RAID definitely out perform MDADM, especially on scrubbing and checksums. I gave up using MDADM on the lowest layer, and opt to use ZFS also. A bonus will be one less software (MDADM) to maintain. I was considering between the two designs below.
| ZFS on LUKS on ZFS: | ZFS (above) <-top
|
LUKS (whole "disk", detached header)
|
ZFS (below) mirror on two disks
|
Raw disk x2 <- bottom |
In this design, I am using ZFS (below) to mirror the data. Both disks are having the same bytes, except the ZFS metadata.
| ZFS on LUKSs: | ZFS mirror <-top
|
LUKS (whole disks, two disks individually, detached headers)
|
Raw disk x2 <- bottom |
This is actually the "comparing to" design in the previous post. The (sole) ZFS is mirroring the data, then feed into both LUKS containers. Although the data are the same, as well as I am using same passphrase for both containers, because of salting, the bytes in both disks are different. One bonus is that total deniability is archived. Not even present of any metadata, there is zero evidences to a present filesystem.
However, managing this design is a little bit harder. First we have to alter the initramfs such that it unlocks both LUKS containers with only one entry of the passphrase. In addition, there will be two detached headers, compared to only one in the "ZFS on LUKS on ZFS" design, that we need to manage them in the key-usb.
Performance-wise I thought the "ZFS on LUKS on ZFS" design would be faster as LUKS only need to encrypt once. However it turns out "ZFS on LUKSs" outperforms in all tests ranging from tiny files to huge files. On huge files, the difference is negatable. On tiny files of 1K, "ZFS on LUKSs" is around 33% faster.
I also tried disabling all features of the ZFS (below) by zpool create -d. It makes no performance differences.
One interesting point to note is that, ZFS is quite memory hungry. On setup of "ZFS on LUKS on ZFS", around 6GB RAM might be reserved for ZFS. When storing tiny files (1K) on disk, the overhead disk space used is quite huge too. However for files from 1M to 100M, the overheads is negatable. On setup of "ZFS on LUKSs", it virtually does not reserve noticeable amount of RAM. RAM is freed after the LUKS computation and data is written to disk.
All in all, "ZFS on LUKSs" is preferable in any aspects.
_________________
- midnite. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
