Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
SATA port multiplier performance
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Mon Feb 22, 2021 7:20 am    Post subject: SATA port multiplier performance Reply with quote

Curious what you guys are seeing with SATA port multipliers.

I had a port multiplier card and finally found a SATA card (sata_sil24) that works with port multipliers that has a regular sata connector, since I can't find my eSATA cable (ugh. wish my ICH* worked, but nope!). The SATA card is a PCIE x1 card...which is surely some sort of bottleneck. It supposedly is a SATA 3.0Gb port.

Anyway I tried running an old RAID6 I set up with discrete SATA ports and got about 120MB/sec sequential reads on a hodgepodge of 500GB rust spinners on the MD device. Eh.. not great but okay...

Currently reformatting RAID5 to see if it's any different...

What kinds of speeds should I expect, I might have to reconfigure to RAID0 to see what it does? Should I be getting near the limit of PCIe x1 of ~250MB/sec? I suspect the jmb363(AHCI) onboard sata controllers I have (with eSATA connector that I don't have a cable for) are also attached x1 and likely similar in performance to the sata_sil24 ...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2990
Location: Canada

PostPosted: Mon Feb 22, 2021 7:37 am    Post subject: Re: SATA port multiplier performance Reply with quote

eccerr0r wrote:
Curious what you guys are seeing with SATA port multipliers.

I had a port multiplier card and finally found a SATA card (sata_sil24) that works with port multipliers that has a regular sata connector, since I can't find my eSATA cable (ugh. wish my ICH* worked, but nope!). The SATA card is a PCIE x1 card...which is surely some sort of bottleneck. It supposedly is a SATA 3.0Gb port.

Anyway I tried running an old RAID6 I set up with discrete SATA ports and got about 120MB/sec sequential reads on a hodgepodge of 500GB rust spinners on the MD device. Eh.. not great but okay...

Currently reformatting RAID5 to see if it's any different...

What kinds of speeds should I expect, I might have to reconfigure to RAID0 to see what it does? Should I be getting near the limit of PCIe x1 of ~250MB/sec? I suspect the jmb363(AHCI) onboard sata controllers I have (with eSATA connector that I don't have a cable for) are also attached x1 and likely similar in performance to the sata_sil24 ...



I tried my back up drives which sit on esata cable, attached to internal header of ICH9 (if I am not mistaken), and it gives 149 MB/sec reads, but alas, it is RAID1, so it is harddrive limited performance, so this is not an answer.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Mon Feb 22, 2021 7:58 am    Post subject: Reply with quote

You have an ICH9R or something like that? I thought ICH9's and ICH10s (non-R) do not support port multipliers. My ICH10 I recall being able to access multiple 2T disks at the same time at full rate (> ~120MB/sec?) and sequential RAID read rate is close to the aggregate bandwidth of the disks. Unfortunately they want to have nothing to do with the port multiplier as I don't have an ICH10R.

The individual disk performance of the 500G disks I have are nothing to write home about, in fact, dismal compared to 149MB/sec - I think some are from 70MB/sec to just about 100MB/sec a piece (mix of WD's and Hitachis). Wondering what I should expect through the port multiplier to a SATA 3Gb port on an PCIe x1...

Yeah I think this array will ultimately be part of my backup solution -- these 500G disks are not enough to backup my 2T disks (which are RAID5'ed) but at least part of the array can be backed up.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2990
Location: Canada

PostPosted: Mon Feb 22, 2021 11:21 pm    Post subject: Reply with quote

eccerr0r wrote:
You have an ICH9R or something like that? I thought ICH9's and ICH10s (non-R) do not support port multipliers. My ICH10 I recall being able to access multiple 2T disks at the same time at full rate (> ~120MB/sec?) and sequential RAID read rate is close to the aggregate bandwidth of the disks. Unfortunately they want to have nothing to do with the port multiplier as I don't have an ICH10R.

The individual disk performance of the 500G disks I have are nothing to write home about, in fact, dismal compared to 149MB/sec - I think some are from 70MB/sec to just about 100MB/sec a piece (mix of WD's and Hitachis). Wondering what I should expect through the port multiplier to a SATA 3Gb port on an PCIe x1...

Yeah I think this array will ultimately be part of my backup solution -- these 500G disks are not enough to backup my 2T disks (which are RAID5'ed) but at least part of the array can be backed up.


ICH9R indeed as far as I see from lspci

Code:

00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)


but the manual for my motherboard, if I am not mistaken with the model, Gigabyte GA-EX58-UD5P/GA-EX58-UD5, shows ICH10R. But maybe the manual (downloaded recently) shows some later revision or something
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Mon Feb 22, 2021 11:31 pm    Post subject: Reply with quote

Mind trying hdparm both disks at the same time and take the aggregate? I wonder what that would do with the interface...

I took my system apart as it's physically very unstable. Need to build an enclosure that will safely house the disks before I can do some more experimentation with it, as it stands it's too easy to jar a disk, which could cause damage to both disks...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2990
Location: Canada

PostPosted: Tue Feb 23, 2021 7:32 am    Post subject: Reply with quote

eccerr0r wrote:
Mind trying hdparm both disks at the same time and take the aggregate? I wonder what that would do with the interface...

I took my system apart as it's physically very unstable. Need to build an enclosure that will safely house the disks before I can do some more experimentation with it, as it stands it's too easy to jar a disk, which could cause damage to both disks...



Unfortunately, I have hardware RAID-1 on external disks in enclosure, so I do not see them separately, it comes to me as a single disk.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Tue Feb 23, 2021 9:12 am    Post subject: Reply with quote

Oh... then that's not a port multiplier then (versus a hardware RAID box)? Perhaps it has PM versus RAID mode?

Port multipliers usually expose all disks to the host directly, or at least the one I have is JBOD and depends on md, lvm, or btrfs to deal with them as a RAID.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 47613
Location: 56N 3W

PostPosted: Tue Feb 23, 2021 10:59 am    Post subject: Reply with quote

eccerr0r,

Port multipliers time division multiplex the drives to the single SATA port so that the host can't tell.
Speed wise, you get whatever the SATA port is capable of as its uninformed of which drive the transfer is actually to/from.
Depending on the drives, you may or may not see a speed improvement overall.

Raid on a SATA port multiplier is scary, since each drive will be written sequentially.
Maybe if you also use a UPS?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Tue Feb 23, 2021 3:20 pm    Post subject: Reply with quote

Well that's what one would expect that it's TDM the multiple drives, but there is dead time between packets due to service latency and this could be used to service another drive, so wondering how that would end up panning out increasing potential bandwidth. Unless the multiplexing forces one to wait out latency (and disables NCQ?), then having a host bandwidth higher than disk bandwidth makes no sense ... but don't know about the actual implementation.

Yeah it would indeed need to be sequential and thus kind of risky for RAID, though it's the same way for any RAID setup that loses power. Speaking of which, old SCSI buses necessarily also wrote sequentially too, so could it be any worse?

Incidentally while mdadm --create the RAID5, during the resync my computer is also showing ~130MB/sec bandwidth over the 5 disks, and the sync is happening at 25MB/sec... so something really is bottlenecked. I suspect that every 25MB/sec written parity to one disk, it needs to read 100MB/sec data from the other four disks. This is clearly pretty bad in terms of using this for RAID5 resync, the direct to motherboard link is much faster as it can read and write each port at full speed at the same time.

I wonder if it's the PM, disks, or a combination that's the specific bottleneck.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Fri Feb 26, 2021 12:49 am    Post subject: Reply with quote

Speaking of these things, I've noticed my silicon image PCIe x1 SATA host adapter automatically detects a hot plugged SATA/port multiplier, but not sure how it's supposed to deal with a hot unplug.

When I want to hot unplug, I unmount and disable LVM on the volumes. Then I pull the plug - but if I lsscsi the drives still show up(!) Something seems to be wrong with hot unplug? Hot plug works beautifully.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
steve_v
Apprentice
Apprentice


Joined: 20 Jun 2004
Posts: 207
Location: New Zealand

PostPosted: Fri Feb 26, 2021 8:13 am    Post subject: Re: SATA port multiplier performance Reply with quote

eccerr0r wrote:
Curious what you guys are seeing with SATA port multipliers.

The first and last time I tried one, performance was awful, compatibility was sketchy, and reliability was non-existent.

If you need more SATA ports, save yourself the pain and just go buy a half-decent PCIE HBA.

eccerr0r wrote:
The SATA card is a PCIE x1 card...which is surely some sort of bottleneck.
Indeed. Expect various levels of disappointing performance, relative to which generation PCIE you're talking about and how many drives are sharing that bandwidth.

Unless one intends to run mechanical disks from 2004, I personally consider anything less than 1 PCIE lane per 2 drives a potential problem.
That's just me of course, and I'm kinda fussy... But any non-shoebox motherboard will likely have a free x8 or x16 slot for that second GPU you probably don't have, so why not put it to better use?

I went there once, doing the cheap PCIEx1 SATA cards (with and without multipliers) thing. Frankly I'd not wish the experience on anyone.
Then I bought a used IBM ServeRAID M1015 (AKA LSI SAS2008) and a SATA breakout cable for 3/5 of bugger all, flashed it to IT mode, and everything was right with the world. I'm still running it ~10 years later and it's still just lovely. I can vouch for the advertised ~350MB/s/port too.

For other recommendations, this list is very much still relevant, or hit up the likes of servethehome/freenas/unraid etc. forums.
_________________
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Fri Feb 26, 2021 3:50 pm    Post subject: Reply with quote

The SATA drives were meant to be offline external backup storage, and speed isn't too big an issue. It actually isn't too bad for single drives it seems, getting close to full performance. I was curious of multiple drive performance which is up there but not quite what I expect. Still not horrible, but didn't quite reach theoretical limits of what I'd think should be possible (that RAID5 resync over SATA/PM I thought should have occurred about 2x as fast as it did, wasn't expecting miracles but it seemed a little low).

Just wanted to reduce number of wires to connect to the drives to deal with the offline nature. Two is good, 6 is not.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
steve_v
Apprentice
Apprentice


Joined: 20 Jun 2004
Posts: 207
Location: New Zealand

PostPosted: Sat Feb 27, 2021 4:20 am    Post subject: Reply with quote

eccerr0r wrote:

I was curious of multiple drive performance which is up there but not quite what I expect.

If your HBA and PM both support FIS, performance won't be too horrible... You'll still suffer on things that need concurrent access to multiple drives of course - like RAID resyncs.
Of bigger concern to me running RAID over a port multiplier is the behaviour under fault conditions. If one drive locks up the bus, it'll lock up the bus for all the drives sharing that port and down goes your array. It's even worse if the drives don't support TLER, in which case individual drive failure of any kind, even just a gnarly URE, is likely to lock up the whole shebang long enough to take the array down.
What you get running RAID over port multipliers is AID, you've created a great big SPOF and your "R" goes down the crapper.

eccerr0r wrote:
Just wanted to reduce number of wires

That's why I like those mini-SAS connectors you find on real HBAs - one cable becomes 4 full-bandwidth SATA ports (that and SAS expanders do what port multiplers do, with fewer issues). You'd still need 3 cables for 6 drives though.
_________________
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18838

PostPosted: Sat Feb 27, 2021 5:27 am    Post subject: Re: SATA port multiplier performance Reply with quote

steve_v wrote:
I bought a used IBM ServeRAID M1015 (AKA LSI SAS2008) and a SATA breakout cable for 3/5 of bugger all, flashed it to IT mode, and everything was right with the world. I'm still running it ~10 years later and it's still just lovely. I can vouch for the advertised ~350MB/s/port too.
Does it require megaraid? I've never encountered a worse CLI interface than that.
_________________
Magna Carta (1215) | Spectral evidence no longer permissible (c. 1792) | Cancel culture, deplatforming (c. 2016)
Back to top
View user's profile Send private message
steve_v
Apprentice
Apprentice


Joined: 20 Jun 2004
Posts: 207
Location: New Zealand

PostPosted: Sat Feb 27, 2021 5:30 am    Post subject: Re: SATA port multiplier performance Reply with quote

pjp wrote:
Does it require megaraid?
Only if you want to run it as a hardware RAID card, which you almost certainly don't. Once flashed to IT mode it's just a dumb HBA with no special tools required, and appears to the OS as JBOD. If you don't need to boot from it you can even nuke the RAID oprom entirely and shave a few seconds off boot time. :)

The LSI CLI tools are indeed horrible, thankfully you only need to use them once, if at all.

Oh, and on the topic of
eccerr0r wrote:
external backup storage

IIRC there are SAS2008 based cards around with external mini-SAS port(s) too, for extra convenience and minimal cabling to your external cage.
Many of those cages even have SAS expander backplanes, and since SAS was actually developed with this in mind (as opposed to (s)ATA multiplexing hacks), you can happily hang 8 spinners off a single SAS cable and still get a reliable ~175MB/s per drive. :D
_________________
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7855
Location: almost Mile High in the USA

PostPosted: Sat Feb 27, 2021 9:56 pm    Post subject: Reply with quote

For free? :-p

Just working with items I happened to have, for curiosity sake. No sense in buying more equipment, not like I manage a datacenter ... err.. well I do for my home network, not for a megacorp paying me to manage a datacenter...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18838

PostPosted: Sun Feb 28, 2021 1:04 am    Post subject: Re: SATA port multiplier performance Reply with quote

steve_v wrote:
pjp wrote:
Does it require megaraid?
Only if you want to run it as a hardware RAID card, which you almost certainly don't. Once flashed to IT mode it's just a dumb HBA with no special tools required, and appears to the OS as JBOD. If you don't need to boot from it you can even nuke the RAID oprom entirely and shave a few seconds off boot time. :)
Good to know, thanks. Sourcing reasonably priced parts from reputable sources seems to continue to be a problem. Some day.
_________________
Magna Carta (1215) | Spectral evidence no longer permissible (c. 1792) | Cancel culture, deplatforming (c. 2016)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum