3ware 9500S + WD Raptor = Whoa!

[jthompson]

I am attempting to setup a box for my company to be a server for simple desktop apps and files, etc.
They gave me a bit of leeway, but told me not to spend too much money.

So not being a total disk I/O expert, I purchased a dual opteron box with a 3Ware 9500S (8 port model) with 6 74 GB WD Raptor disks.
Its quite possible, if I ever get it fully up and running and "live" that it could have as much as 100 users (I'm not sure yet).

Anyway, before I do a final install (I've done a GNOME install and decided that KDE would better suit my needs)- so I thought I would read up on 3Ware and the 2.6 kernel driver. I have not used the kernel source from the website, I just used the default driver already in a 2.6.9-r9 kernel.

If I configure a RAID 10 array with a stripe size of 16 KB with six drives I get the following from hdparm -tT /dev/sda

Code: Select all

Timing cached reads: 2920 MB in 2.00 seconds = 1460.22 MB/sec
Timing buffered cached reads: 334 MB in 3.01 seconds = 110.94 MB/sec

I noticed that when I configured an array and I increased the stripe size, that the performance degraded significantly with the default options.

However, I have noted that many of the options in hdparm relate to IDE/ATA, which in the kernel the 3Ware controller is listed as a SCSI device. So I thought, there must be another utitlity or option that exists somewhere that can change the performance of SCSI devices?

I started poking around on 3Ware's website and found this.

http://www.3ware.com/KB/kb.asp
Bah I just realized this link does not take you directly to the article, do a search for article # Q11050 in 3ware's Knowledge base.
https://www.3ware.com/kbadmin/attachmen ... P_0701.pdf

Basically it says they are investigating it and lists a whole bunch of tests you can do, which I have yet to do.

However, I did use the blockdev command with the value they suggested.

Code: Select all

blockdev --setra 16384 /dev/sda

And I got this from hdparm

Code: Select all

Timing cached reads:   2948 MB in 2.00 seconds = 1472.75 MB/sec
Timing buffered disk reads: 640 MB in 3.00 seconds = 213.01 MB/sec

You have to be kidding, it doubled?

So basically I pose these questions...
What have you folks found with your 3Ware controllers?
Have you tried different benchmarks?
What settings can you change for a SCSI disk, even though this really is not a SCSI disk?
Post your findings here.

[mattst88]

/me begins to drool

[svan9420]

Those are some pretty good bench marks.

[JetAce44]

My god... I just nearly doubled my performance on my raid 5 array by using the blockdev command suggested.

Went from:

Code: Select all

BlackBox # hdparm -tT /dev/sda

/dev/sda:
 Timing buffer-cache reads:   1004 MB in  2.00 seconds = 500.82 MB/sec
 Timing buffered disk reads:  106 MB in  3.01 seconds =  35.26 MB/sec

To:

Code: Select all

BlackBox # blockdev --setra 16384 /dev/sda
BlackBox # hdparm -tT /dev/sda

/dev/sda:
 Timing buffer-cache reads:   1088 MB in  2.00 seconds = 543.00 MB/sec
 Timing buffered disk reads:  180 MB in  3.02 seconds =  59.63 MB/sec

:drool:

Thanks for the tip!

This is on the hardware in my sig, btw.

[PaulMezz]

Configuration:
Two 7850-8 controllers, each with 8 Maxtor 4G120J6 drives (120g, 5400rpm)
3 arrays, Controler 0 has 2 Raid-5 of 4 drives each (sda/sdb)
Controller 1 has all 8 drives in a Raid-10 (sdc)

All using defaults when created.
(sda and sdb are the same, for the sake of saving space I am only posting sda)

Code: Select all

uber root # hdparm -tT /dev/sda

/dev/sda:
 Timing buffer-cache reads:   2424 MB in  2.00 seconds = 1211.58 MB/sec
 Timing buffered disk reads:   76 MB in  3.02 seconds =  25.19 MB/sec

uber root # blockdev --setra 16384 /dev/sda
uber root # hdparm -tT /dev/sda

/dev/sda:
 Timing buffer-cache reads:   2452 MB in  2.00 seconds = 1225.57 MB/sec
 Timing buffered disk reads:  244 MB in  3.00 seconds =  81.21 MB/sec

56.02 MB/sec increase, AKA a 222% increase

The Raid-10 increase is not as impressive

Code: Select all

uber root # hdparm -tT /dev/sdc

/dev/sdc:
 Timing buffer-cache reads:   2372 MB in  2.00 seconds = 1186.18 MB/sec
 Timing buffered disk reads:  322 MB in  3.01 seconds = 107.14 MB/sec

uber root # blockdev --setra 16384 /dev/sdc
uber root # hdparm -tT /dev/sdc

/dev/sdc:
 Timing buffer-cache reads:   2376 MB in  2.00 seconds = 1188.18 MB/sec
 Timing buffered disk reads:  366 MB in  3.02 seconds = 121.33 MB/sec

14.19 MB/sec increase, AKA a 13% increase. Not bad for free

jthompson, I owe you a beer.

-paul

[tam]

This is my Raptor 74 on a nforce4 ultra

Code: Select all

amd64 ~ # hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   3204 MB in  2.00 seconds = 1601.44 MB/sec
 Timing buffered disk reads:  186 MB in  3.01 seconds =  61.70 MB/sec

AMD64 3200+ Winchester, gentoo-sources 2.6.12-r4

[SilverOne]

These were my results (booting from Gentoo-amd64 livecd obviously this system is as near to idle as possible)

Code: Select all

livecd root # blockdev --getra /dev/sda
256
livecd root # hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   4456 MB in  2.00 seconds = 2227.22 MB/sec
 Timing buffered disk reads:  238 MB in  3.00 seconds =  79.29 MB/sec
livecd root # blockdev --setra 16384 /dev/sda
livecd root # hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   4484 MB in  2.00 seconds = 2240.10 MB/sec
 Timing buffered disk reads:  304 MB in  3.03 seconds = 100.35 MB/sec
livecd root # hdparm -tT /dev/sda

I re-ran the tests several times and I get pretty consistent results.

Machine specs:
AMD64 Opteron 2.2 GHz
2GB ram
4xWestern Digital 200GB drives in a single RAID 5 array

I was pretty impressed The 25% performance increase is very nice too!
Though not as good as jthompson yet

Edit: the drives are attached to a 3ware 8506 controller (in a regular 33MHz PCI slot I might add)

[jsnorman]

Holy cr*([p. I have an Athlon 53 FX system at home running a RAID-5 array on the 9500S using the kernel drivers in 2.6.12-gentoo-r6. As soon as I get home I am going to try this and will report back. I didn't even think there was much tweaking possible since the drivers are in the kernel now. Wow.

[JetAce44]

Anyone else getting extreme system lag and high loading when doing intensive I/O tasks? I get about .5-2 second hangups and pauses when doing I/O on the disk with large file deletions and creations ever since implimenting this tweak. Anyone know of a way to undo the setting?

[SilverOne]

That depends on what the original setting was.

On my system the original ReadAhead setting (the one we're tweaking here) was set to 256, as I figured out using:

Code: Select all

livecd root # blockdev --getra /dev/sda 
256

Before running the tweak.

I suppose if your system had the same starting value, you could reset it by issuing:

Code: Select all

livecd root # blockdev --setra 256 /dev/sda

(replacing 256 and sda as necessary)

NOTE: I have no idea what a really wrong setting could do to your system so, follow the advice at your own risk

[jsnorman]

Improvement but not as much as others:

tux jsnorman # hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 2688 MB in 2.00 seconds = 1341.52 MB/sec
Timing buffered disk reads: 186 MB in 3.02 seconds = 61.64 MB/sec
tux jsnorman # hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 2852 MB in 2.09 seconds = 1362.84 MB/sec
Timing buffered disk reads: 278 MB in 3.00 seconds = 92.59 MB/sec

I am not using raptors, but I do have the 9500S. Running kernel 2.6.12-gentoo-r6. My drives are Maxtor Diamonmax 10 (with NCQ) 250GBx3 in RAID 5. They are used only for data. My / drives are using software raid (md) but I did not test them yet.

[fbvortex]

What is the recommended way to make the read-ahead setting configured by "blockdev --setra" persistent across boots / reboots?

Thanks in advance.

[jschellhaass]

You could use /etc/conf.d/local.start.

jeff

[jthompson]

You could use /etc/conf.d/local.start.

In regards to making it happen if you do a reboot. I think this is the only place to put it, that I know of.

Code: Select all

/etc/conf.d/local.start

blockdev --setra 256 /dev/sda

I believe this is the default setting before you change anything. Just change it back or try different values if you have problems.

The funny thing is I have two identical servers at work, one used for storage, one used for a number of other things (mainly desktop connections, etc).

They generate two totally different values.

Server 1 (2.6.10-r3 hardened kernel) - I don't use any of the hardened features, I just use it because Gentoo says to use that one for servers.

Code: Select all

/dev/sda:
 Timing cached reads:   2580 MB in  2.00 seconds = 1288.91 MB/sec
 Timing buffered disk reads:  394 MB in  3.01 seconds = 130.74 MB/sec

That is with 10 or more users on the system though. The highest I've ever got with this machine is 150 MB/sec.

The backup/storage machine, ironically, gets quite a bit more.

Server 2 (2.6.11-r1-hardened)

Code: Select all

/dev/sda:
 Timing cached reads:   2956 MB in  2.00 seconds = 1477.49 MB/sec
 Timing buffered disk reads:  742 MB in  3.00 seconds = 247.12 MB/sec

No users connect to this system.

Maybe they updated the drivers in the 2.6.11 kernel, I dunno. They both have a 9500S with 6 Raptors, with a RAID 10 array. Its kind of depressing because I would rather the machine that users connect to be faster, but oh well. When I have time, I will try and update to the latest kernel. I'm just kind of scared too, as they are in use all the time. Plus I wonder if I can use the .config file from a 2.6.10 to get started on a 2.6.11 or 2.6.12?

I've also updated the bios on the mainboards and the firmware on the 3ware controllers a time or two, which may have also helped. But you have to reboot to do that and take the machine down. Glad other folks are having success with it!

JetAce44, I would say try updating the bios of the motherboard (then load theload setup defaults), then update the firmware of the 3ware card. You can do it from floppy.

http://www.3ware.com/support/download.asp

If that doesn't work, try different values and see what happens. 3ware admits they need further testing with the 2.6 kernel to find all the ins and outs. I found this info in their knowledge base.

http://www.3ware.com/KB/kb.asp?page=search

[bmouring]

Been using my 9500S with dual raptors as the root/boot device for a few weeks now, stable and screamin'.

@ JetAce, haven't really noticed any lag/delaying even during moving my music from another disk to this array... maybe just been lucky? In any case, jthompson's advice is solid for any device that accepts BIOS/firmware updates, but one caveat is that there have been times when I have seen a perfectly good device become crippled/degraded with a new firmware, so the old adage "If it ain't broke.." still applies. That said, it definately seems like there is little to lose in this situation.

On boot, using local.start (along with some key mapping stuff), loading the RA values (with hdparm instead, dunno why, just the way I found out about it) consistently get ~130MB/s.

[RockHound]

Amazing ... RAID 50 on a 3ware 9500S-12

before blockdev:

Code: Select all

/dev/sdb:
 Timing cached reads:   2328 MB in  2.00 seconds = 1163.93 MB/sec
 Timing buffered disk reads:  154 MB in  3.01 seconds =  51.13 MB/sec

after blockdev:

Code: Select all

/dev/sdb:
 Timing cached reads:   2304 MB in  2.00 seconds = 1151.93 MB/sec
 Timing buffered disk reads:  904 MB in  3.01 seconds = 300.51 MB/sec

Hmm ... thanks for the hint! Is the blockdev command a line for /etc/conf.d/local.start?? or is it permanent now?

[nianderson]

Amazing ... RAID 50 on a 3ware 9500S-12

before blockdev:

Code: Select all

/dev/sdb:
 Timing cached reads:   2328 MB in  2.00 seconds = 1163.93 MB/sec
 Timing buffered disk reads:  154 MB in  3.01 seconds =  51.13 MB/sec

after blockdev:

Code: Select all

/dev/sdb:
 Timing cached reads:   2304 MB in  2.00 seconds = 1151.93 MB/sec
 Timing buffered disk reads:  904 MB in  3.01 seconds = 300.51 MB/sec

Hmm ... thanks for the hint! Is the blockdev command a line for /etc/conf.d/local.start?? or is it permanent now?

I believe it needs to be in local.start

What model drives do you have?

[mephist0]

I have a 3ware 9650SE PCI-Express controller with 2TB RAID-5 on it (8 Western Digital 320GB)

hdparm is just amazing

thanks for the blockdev hint

Code: Select all

hdparm -tT /dev/sdc

/dev/sdc:
Timing cached reads:   15180 MB in  2.00 seconds = 7596.94 MB/sec
Timing buffered disk reads:  886 MB in  3.00 seconds = 295.30 MB/sec

but when I extract or test .rar files I only get 40-50 MB/sec
I encrypted the RAID with cryptsetup-luks

Is rar limited to CPU speed ?

Is the encryption slowing transfers down ?

thanks for a hint !

[mephist0]

*push*

this is my dmesg output:

Code: Select all

3ware 9000 Storage Controller device driver for Linux v2.26.02.008.
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:02:00.0 to 64
scsi7 : 3ware 9000 Storage Controller
3w-9xxx: scsi7: Found a 3ware 9000 Storage Controller at 0xe7001000, IRQ: 16.
3w-9xxx: scsi7: Firmware FE9X 3.06.00.003, BIOS BE9X 3.06.00.002, Ports: 8.
scsi 7:0:0:0: Direct-Access     AMCC     9650SE-8LP DISK  3.06 PQ: 0 ANSI: 5
sdc : very big device. try to use READ CAPACITY(16).
SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB)
sdc: Write Protect is off
sdc: Mode Sense: 23 00 00 00
SCSI device sdc: drive cache: write back, no read (daft)
sdc : very big device. try to use READ CAPACITY(16).
SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB)
sdc: Write Protect is off
sdc: Mode Sense: 23 00 00 00
SCSI device sdc: drive cache: write back, no read (daft)
 sdc: sdc1
sd 7:0:0:0: Attached scsi disk sdc

I uses parted to create the partition with a GPT table

During transfer the system kinda freezes for 0.5-1 secs several times and then writes the data with ~150-250MB/sec
The transfer is stuck and then it writes the data
and if I move the mouse or type anything with the keyboard this also "freezes" and after 0.5-1 secs it moves the cursor and/or the chars are written ...

I dunno why this is

I had a 7506-4Port 3ware ATA controller and this didnt happen at all

I use kernel 2.6.19-gentoo-r4

I cant use 2.6.18.* kernels, because there is no driver for the 9650 cards

I dont know if its a kernel or controller or config error since there is no error message

I hope anyone can give me a hint!

here is my complete dmesg output:

http://rafb.net/p/NMPTIn47.html

here is my kernel .config:

http://rafb.net/p/rfBPu415.html

here is my lspci:

Code: Select all

00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02)
00:01.0 PCI bridge: Intel Corporation 82P965/G965 PCI Express Root Port (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7800 GTX] (rev a1)
02:00.0 RAID bus controller: 3ware Inc Unknown device 1004 (rev 01)
03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 22)
04:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)
04:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)
05:00.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)
05:06.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)

[mephist0]

*bump*

nobody ?

this is *REALLY* urgent

cant use such a system that freezes every 10 secs during transfer on the raid

please, somebody !!!

I sent a support ticket to 3ware but they didnt answer yet

[valmiki]

*bump*

nobody ?

this is *REALLY* urgent

cant use such a system that freezes every 10 secs during transfer on the raid

please, somebody !!!

I sent a support ticket to 3ware but they didnt answer yet

Is there a firmware update for it?

[mephist0]

*bump*

nobody ?

this is *REALLY* urgent

cant use such a system that freezes every 10 secs during transfer on the raid

please, somebody !!!

I sent a support ticket to 3ware but they didnt answer yet

Is there a firmware update for it?

Yes, latest firmware is installed

I just switched back to kernel 2.6.18-gentoo-r6

The "freezing" is gone, but still only max 50-60MB/sec read and write performance

Isnt that kinda slow ?

[simeli]

read my posts over here: http://forums.gentoo.org/viewtopic-p-3949657.html

what filesystems are you guys using?