3ware raid-5 performance issues

Message

kinkozmasta · Post by **kinkozmasta** » Sat Jan 27, 2007 2:01 am

Hi,

Posted a few days ago about problems getting my 3ware 9650SE RAID card up and running. I finally did but now I am having some very strange performance issues with it, that maybe somebody can shed some light on. I've set it up with 3 320GB SATA II drives in RAID-5.

Here's the configuration:

Code: Select all

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    OK             -       -       64K     596.025   OFF    OFF

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     298.09 GB   625142448     9QF1GP48
p1     OK               u0     298.09 GB   625142448     9QF1F2XY
p2     OK               u0     298.09 GB   625142448     9QF1F0WE
p3     NOT-PRESENT      -      -           -             -

Read performance seems to be pretty good but write performance is terrible and very sporadic. Copying large files or groups of large files (either from a different drive to the array or from the array back to the same array) takes forever. What is particularly strange (to me at least) is that there will be bursts when it is incredibly fast and then go right back to a snails pace. For example, often when copying a 1GB the first 300MB will copy in just a few seconds but then it will slow or even stop, then speed up a little (but usually only just a little to 5-10MB/s) then back down again. While copying a directory with lots of medium sized files in the 5-20MB range something similar happens.
Also while writing the system as a whole becomes unresponsive and "jerky".

I've tried enabling the cache even though it is not recommended without the battery backup unit, but that didn't help (so I turned it back off).
I'm running the gentoo-sources kernel 2.16.19-r4 and using the driver in the kernel.
Preemption is also compiled into the kernel.

This is on an Athlon64 X2 4400+.

Did I forget to mention anything useful?

Any ideas?

ksp7498 · Post by **ksp7498** » Sat Jan 27, 2007 2:25 am

I had an issue that sounds very similar to this a while back, though I was not running a raid array. My synthetic transfer rates (hdparm -tT) were normal but actual transfer rates when copying in nautilus were extremely slow. Turns out it was a bug in Nautilus when using XFS filesystems.

What filesystem are you using, and what program are you using to copy files?

kinkozmasta · Post by **kinkozmasta** » Sat Jan 27, 2007 3:10 am

I'm running ext3 on the array. I first noticed the bad transfer rates when trying to copy in a terminal using cp. However, I noticed the fluctuating performance when I rebooted into KDE and transfered some big files using Konqueror.

I forgot to mention that I have two other (SATA) hard drives installed too, if that makes any difference.

Ast0r · Post by **Ast0r** » Sat Jan 27, 2007 3:41 am

3 drives in RAID5 is a bad idea. Write performance is going to suck the big one. You need 4 or 5 drives to get decent performance.

You could try setting the kernel's read-ahead buffer (see this thread), but I doubt it's going to help your write performance much (although it will almost certainly help read performance ... about 500% on my systems). One thing that will almost certainly help is turning the write cache on for the card. I forget the command to do it, but it's in the 3ware manual somewhere. According to the output you pasted, it's off on your card. Be forewarned that without the BBU it's not totally "safe" and I've had an array corrupt because of a power loss when I had it on (although it was a trivial process to fix it), but as long as your machine is on a good UPS, you should probably be ok.

kinkozmasta · Post by **kinkozmasta** » Sat Jan 27, 2007 4:08 am

Astor,
I don't think those things are the problem. I tried enabling the write cache and it basically made no difference.
Also, on say a typical 1GB file, the first half usually writes at 100MB/s for the first 500MB and then just screeches to a halt. Then kind of fluctuates. If what you said were true I would expect the performance to be consistently low, say write at 20-30MB over the whole file.

I have also tried the things in the thread you mentioned but they made no difference. Actually the results I get from hdparm are quite good.

Code: Select all

redleader ~ # hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   3600 MB in  2.00 seconds = 1800.22 MB/sec
 Timing buffered disk reads:  450 MB in  3.00 seconds = 149.93 MB/sec

there are also these optimizations suggested on 3ware's site for the 9650

echo 64 > /sys/block/sda/queue/max_sectors_kb
echo deadline > /sys/block/sda/queue/scheduler

but again they don't really help. Actually changing the scheduler makes the responsiveness to the system in general worse and does not significantly improve the write performance.

mephist0 · Post by **mephist0** » Tue Feb 13, 2007 6:23 am

hi

I have similar problems

I have a 9650SE-8port card.

8 Western Digital 320GB -> RAID-5

hdparm -tT gives me 430MB/sec buffered reads.

but with bonny++ (6GB file) I get only max 50MB/sec read and write!!!

I use kernel 2.6.20

I tried kernel 2.6.18, .19 and .17 but no difference ....

here is my post

thanks for your help

I am at work right now, the can post the current kernel .config as soon as I get home ...

simeli · Post by **simeli** » Fri Mar 09, 2007 10:56 am

i read a post (i believe on the 3ware site) that picked up a similar problem. it seems that this is heavily filesystem dependant. ext3 scores badly in those tests whereas xfs is a real screamer. i'll try to look up the article with the benches. in the meantime:

http://www.3ware.com/LInuxSell_0629.pdf
http://www.issociate.de/board/post/2370 ... mance.html

simeli · Post by **simeli** » Fri Mar 09, 2007 11:00 am

this is the article i was referring to:
http://forums.storagereview.net/index.p ... =2&t=24237

Cyker · Post by **Cyker** » Fri Mar 09, 2007 3:37 pm

The initial high-speed bit is probably Linux buffering up the data.

What's this 3ware card attached to, the PCI Bus?

What else is attached to the bus?

I do know that Linux Softwae RAID5 is significantly slower at writing than reading because it has to do all those calculations, but I had thought a pure hardware RAID5 card would be quicker - Perhaps this is not the case?

simeli · Post by **simeli** » Fri Mar 09, 2007 3:50 pm

from what i understand the 9650SE is pci express, is it not?

mephist0 · Post by **mephist0** » Fri Mar 09, 2007 6:23 pm

simeli wrote:from what i understand the 9650SE is pci express, is it not?

Yes it is PCI-Express x4.

I think its the encryption which is slowing the transfers down.

but I dont want to backup 500GB of stuff to reformat and try unencrypted.

but it MUST be the encryption.
because the process "kcryptd" shows 100% cpu usage while transfering (copy, move, extracting .rar files) data.

I tried everything I know off.

But no go...

I gave up on finding a solution a week ago.

but if you have a solution it would be cool

ah, forgot to mention I tried ext3 and reiserfs.
didnt try XFS because I have no UPS or battery backup unit

simeli · Post by **simeli** » Sun Mar 11, 2007 12:16 am

as you can see from the article, xfs seems to be the best performer in that area, sometimes by a wide margin. running it w/o a UPS is bad true. however, i have personally never lost data due to a power outage with xfs. ext3 is sometimes but not necessarely better, because it tries to guess what the correct data must have been. xfs marks these files as corrupt and fills them with nothing. at least you know which ones are bad. aggressively caching data during writes of course gives you a higher chance of losing something. in general though i think it is overrated. just my 2c

mephist0 · Post by **mephist0** » Sun Mar 11, 2007 11:20 am

simeli wrote:as you can see from the article, xfs seems to be the best performer in that area, sometimes by a wide margin. running it w/o a UPS is bad true. however, i have personally never lost data due to a power outage with xfs. ext3 is sometimes but not necessarely better, because it tries to guess what the correct data must have been. xfs marks these files as corrupt and fills them with nothing. at least you know which ones are bad. aggressively caching data during writes of course gives you a higher chance of losing something. in general though i think it is overrated. just my 2c

I dont believe it is the filesystem thats slowing all down.

If I run "top" while running bonnie++ tests, it shows 20-50% I/O wait ("wa" in top is this I think).

I am trying another kernel config now whith the newest gentoo-sources.

and I recently testet my raptor 150GB (reiserfs encrypted with cryptsetup-luks) and it also got ca. 20-50% I/O wait !!!

drdope · Post by **drdope** » Sun Mar 11, 2007 5:50 pm

ksp7498 wrote:I had an issue that sounds very similar to this a while back, though I was not running a raid array. My synthetic transfer rates (hdparm -tT) were normal but actual transfer rates when copying in nautilus were extremely slow. Turns out it was a bug in Nautilus when using XFS filesystems.

I think, i'm having the same problem...or some sort of...

Interestingly its only occuring, when i transfer Data via GB-LAN between Server & Workstation.
--> transfer rates go up (~max 100MB/s) and down (~1kb/s) intermittingly; in average about 30MB/s writing an 20MB/s reading (measures using date && copy && date and doing the math)

If i transfer data locally on the Server, transfer rates are about 65-75MB/s constantly
(my server runs without a gui - Stage3 + NFS/Samba + 3dm2) - , but i'm using Gnome/Nautilus on my Workstation; the Raid5s are formated with XFS and accessed via NFS)

Are there any references to this "nautilus/xfs Bug"?

mephist0 · Post by **mephist0** » Tue Mar 13, 2007 9:20 pm

mephist0 wrote: but it MUST be the encryption.
because the process "kcryptd" shows 100% cpu usage while transfering (copy, move, extracting .rar files) data.

I backup my raid, then mkfs.xfs /dev/sdc1

bonnie++ run

and OH MY GOD !!!!

write 180MBYTE/S
rewrite 90MBYTE/S
read 450MBYTE/S

^^^ without encryption

with encryption (I used this guide: http://gentoo-wiki.com/SECURITY_System_ ... _with_LUKS )

write 50MB/S
rewrite 30MB/S
read 50MB/S

^^^^ WTF ?!?!?

simeli · Post by **simeli** » Wed Mar 14, 2007 7:31 am

q.e.d.