Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Performance problems with mdadm (Raid6) + Crypto Luks + LVM2
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
rralf
n00b
n00b


Joined: 10 Jun 2012
Posts: 9

PostPosted: Sun Nov 18, 2012 2:20 pm    Post subject: Performance problems with mdadm (Raid6) + Crypto Luks + LVM2 Reply with quote

Hi!

I have terrible performance problems with my hard disk setup.

This is my Configuration:
4 x 3TB Sata hard drives (/dev/sd[abcd]) formatted with a GPT Partition Table.

/dev/sd[abcd]1 : 100 MiB Boot Drive (RAID1 without encryption/LVM) <- works fine
/dev/sd[abcd]2: 2.73TiB Data Storage (Raid6) <- problem child

According to gparted, both partitions are aligned.

This is what I did for formatting the data storage:

Raid Setup:
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sd[abcd]2
(15 hours of initialization... ;-) )
Speed of initialization: ~30-50Mib/s <- In my opinion, even this is slow
If I do "dd if=/dev/zero of=/dev/sda2 bs=512K count=1024" I have round about 100MB/s write speed.
Well...


Crypto Setup:
cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --key-size 256 /dev/md0
cryptsetup luksOpen /dev/md0 Storage

LVM2 Setup:
According to /proc/mdstat, chunk size of my Raid6 is 512KiB

pvcreate --dataalignment 512K -M2 /dev/mapper/Storage
vgcreate vg /dev/mapper/Storage
lvcreate -L100G -nStore1 vg
mkfs.ext4 -j -LStore1 /dev/vg/Store1
mount /dev/vg/Store1 /mnt/foobar
Everything works fine except of the performance :-)

Read speed is about ~100MiB/s. That's okay for me.
But write speed is ~20MiB/s, that's really bad.

What am I doing wrong?
I'm sure write speed could be faster than 20Mib/s....

Thanks for help!
Best regards,

Ralf
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 1562
Location: Berlin, Germany

PostPosted: Mon Nov 19, 2012 2:47 am    Post subject: Reply with quote

Do you have the same problem when you create the filesystem directly on /dev/md0 (instead of using lvm)?
Back to top
View user's profile Send private message
rralf
n00b
n00b


Joined: 10 Jun 2012
Posts: 9

PostPosted: Mon Nov 19, 2012 12:37 pm    Post subject: Reply with quote

No, I tried it out.

When i directly create a filesystem on /dev/md0, performance is fine.
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 1562
Location: Berlin, Germany

PostPosted: Mon Nov 19, 2012 12:42 pm    Post subject: Reply with quote

So given that read performance is fine and write performance without LVM is fine too, the problem looks like an LVM alignment issue.
Back to top
View user's profile Send private message
rralf
n00b
n00b


Joined: 10 Jun 2012
Posts: 9

PostPosted: Mon Nov 19, 2012 12:44 pm    Post subject: Reply with quote

Yes, probably in combination with cryptsetup, but i don't know what I'm doing wrong...

I tried to align everything correctly...
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 1562
Location: Berlin, Germany

PostPosted: Mon Nov 19, 2012 12:45 pm    Post subject: Reply with quote

Ah sorry, I misread what you wrote above.

What if you create the filesystem on /dev/mapper/Storage? How is the performance then?
EDIT: cryptsetup does have an --align-payload option, did you try that too?
Back to top
View user's profile Send private message
rralf
n00b
n00b


Joined: 10 Jun 2012
Posts: 9

PostPosted: Mon Nov 19, 2012 3:08 pm    Post subject: Reply with quote

Ok, i read some good articles:

https://support.mayfirst.org/wiki/disk_alignment
http://wiki.drewhess.com/wiki/Creating_an_encrypted_filesystem_on_a_partition
http://blog.kanojo.de/2011/10/11/raid-system-8tb-home-storage-on-budget/

After tweaking and trying i got it a bit better:

echo 32768 > /sys/block/md127/md/stripe_cache_size

This brought an amazing speedup!

Read speed: ~200MB/s, tested with:
dd if=/dev/md127 of=/dev/null bs=512K

Write speed: ~120MB/s
dd if=/dev/zero of=/dev/md127 bs=512K

That's quite good.

Cryptsetup tweaking
cryptsetup luksFormat --argsfoobar --align-payload 4096 /dev/md127
cryptsetup luksOpen /dev/md127 vault
...

Crypto Read speed: ~100MB/s, tested with:
dd if=/dev/mapper/vault of=/dev/null bs=512K

Crypto Write speed: ~90MB/s
dd if=/dev/zero of=/dev/mapper/vault bs=512K

That's not the truth... Read speed is half than before, write speed decreased also by 30MB/s....
But I suppose, that this is the crypto overhead.

LVM Creation:
pvcreate --metadatasize 2048K --dataalignment 4096 -M2 /dev/mapper/vault
vgcreate vg /dev/mapper/vault
lvcreate -nGentoo -L30G vg

LVM Read speed: ~100MB/s, tested with:
dd if=/dev/vg/Gentoo of=/dev/null bs=512K

LVM Write speed: ~90MB/s
dd if=/dev/zero of=/dev/vg/Gentoo bs=512K

So speed doesn't decrease when using LVM. (This is an important information for me....)

After creating the ext4fs using
mkfs.ext4 -j -LGentoo -b 4096 -E stride=128,stripe-width=256 /dev/vg/Gentoo
mount /dev/vg/Gentoo /mnt/gentoo

Ext4 Read speed: ~100MB/s, calculated with:
sync;date;cp /mnt/gentoo/3gbfile /mnt/ramfs/foobar;sync;date

Ext4 Write speed: ~85MB/s, calculated with:
sync;date;cp /mnt/ramfs/3gbfile /mnt/gentoo/3gbfile;sync;date

So the bottleneck seems to be the cryptolayer....

Anyone further ideas?

Thanks!
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 1562
Location: Berlin, Germany

PostPosted: Mon Nov 19, 2012 3:46 pm    Post subject: Reply with quote

What is the read/write speed if you create a luks volume on a single disk?

Also if your CPU does not support AES acceleration, better use 128-bit encryption (aes-xts-plain with 256 bits, half of which are the AES key) and not 256-bit (aes-xts-plain 512).
Back to top
View user's profile Send private message
rralf
n00b
n00b


Joined: 10 Jun 2012
Posts: 9

PostPosted: Mon Nov 19, 2012 3:59 pm    Post subject: Reply with quote

R/W on a single disk: ~70MB/s

Urgh, according to /proc/cpuinfo my CPU doesn't support AES instruction set.

Damn....
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 1562
Location: Berlin, Germany

PostPosted: Mon Nov 19, 2012 4:01 pm    Post subject: Reply with quote

Code:
# openssl speed aes-128-cbc
can tell you how fast you CPU does AES-128. If the CPU does AES-NI hardware acceleration,
Code:
# openssl speed -evp aes-128-cbc
will show you the accelerated speed (requires >=openssl-1.0.1).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum