View previous topic :: View next topic |
Author |
Message |
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Thu Jan 24, 2013 7:43 am Post subject: AsMedia ASM1061 Port Multiplier issues |
|
|
Hi All,
I'm running an Intel D510MO mainboard with has a MiniPCIe slot. According to specs it has a PCIe 1x v2 bus capable of 500MB/s. Currently it has a Silicon Image 3132 based 2-port SATA controller installed, the second SATA port is attached to an Addonics SATA port multiplier. The PM is Sil3726 based. There are 5 3TB WD30EZRX disks attached to the PM stuffed in an mdadm raid5 array.
This works fine but I am not able to get more throughput than approx 130MB/s through the Sil3132. It appears that despite being SATA300 the card/chip has an internal limit that is bottlenecking my disk IO.
After doing some research I found that the Asmedia 1061 chip is very cheap and it is capable of reaching speeds well over 300MB/s. Because I'm running a SATA300 port multiplier I don't expect to make those speeds but at least well over 130MB/s would be nice, 250MB/s would be more in line with my expectation.
I went ahead and got myself one of these: http://www.delock.de/produkte/F_426_Module_95225/merkmale.html
2 port MiniPCIe based SATA controller that uses an Asmedia 1061 chip. Plugged it in and the BIOS shows the card is in AHCI mode, as I already had AHCI SATA enabled in the kernel it was picked up just fine:
Code: |
[ 0.622850] ahci 0000:02:00.0: irq 45 for MSI/MSI-X
[ 0.622924] ahci: SSS flag set, parallel bus scan disabled
[ 0.623107] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
[ 0.623305] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part ccc sxs
[ 0.624803] scsi4 : ahci
[ 0.625285] scsi5 : ahci
[ 0.625691] ata5: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200100 irq 45
[ 0.625889] ata6: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200180 irq 45
|
It finds the disks through the port multiplier:
Code: |
[ 1.948274] ata5.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[ 1.948438] ata5.01: hard resetting link
[ 2.440248] ata5.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.440408] ata5.02: hard resetting link
[ 2.932252] ata5.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.932410] ata5.03: hard resetting link
[ 3.424248] ata5.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 3.424407] ata5.04: hard resetting link
[ 3.916245] ata5.04: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 3.916403] ata5.05: hard resetting link
[ 7.236028] ata5.15: qc timeout (cmd 0xe4)
[ 7.236154] ata5.05: failed to read SCR 1 (Emask=0x4)
[ 7.236269] ata5.05: failed to read SCR 0 (Emask=0x40)
[ 7.237029] ata5.00: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[ 7.237148] ata5.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 7.237910] ata5.00: configured for UDMA/133
[ 7.238611] ata5.01: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[ 7.238732] ata5.01: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 7.239484] ata5.01: configured for UDMA/133
[ 7.240187] ata5.02: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[ 7.240312] ata5.02: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 7.241070] ata5.02: configured for UDMA/133
[ 7.241771] ata5.03: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[ 7.241892] ata5.03: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 7.242648] ata5.03: configured for UDMA/133
[ 7.243350] ata5.04: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[ 7.243472] ata5.04: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 7.244241] ata5.04: configured for UDMA/133
[ 7.244429] ata5: EH complete
|
And after booting the array was properly assembled. "Benchmarking" individual disks looks fine:
Code: |
/dev/sdc:
Timing cached reads: 1574 MB in 2.00 seconds = 787.55 MB/sec
Timing buffered disk reads: 438 MB in 3.01 seconds = 145.53 MB/sec
/dev/sdd:
Timing cached reads: 1540 MB in 2.00 seconds = 770.47 MB/sec
Timing buffered disk reads: 476 MB in 3.00 seconds = 158.51 MB/sec
/dev/sde:
Timing cached reads: 1540 MB in 2.00 seconds = 770.61 MB/sec
Timing buffered disk reads: 426 MB in 3.02 seconds = 141.08 MB/sec
/dev/sdf:
Timing cached reads: 1544 MB in 2.00 seconds = 772.48 MB/sec
Timing buffered disk reads: 424 MB in 3.02 seconds = 140.44 MB/sec
/dev/sdg:
Timing cached reads: 1548 MB in 2.00 seconds = 773.76 MB/sec
Timing buffered disk reads: 428 MB in 3.01 seconds = 142.16 MB/sec
|
The real problem occurs when accessing the array:
Code: |
/dev/md5:
Timing cached reads: 1428 MB in 2.00 seconds = 714.01 MB/sec
Timing buffered disk reads: 70 MB in 3.34 seconds = 20.93 MB/sec
|
In comparison; the Sil3132 does this (while rebuilding actually):
Code: |
/dev/md5:
Timing cached reads: 1258 MB in 2.00 seconds = 629.35 MB/sec
Timing buffered disk reads: 358 MB in 3.01 seconds = 118.80 MB/sec
|
Stumped as I was I ran the hdparm a couple of times to make sure the result was consistent. The results were even worse; sometimes 2MB/s up to 7MB/s. Hmm weird; so to confirm I wrote a 1GiB test file on the array which was very slow. After a minute or so it stalled and the array kinda crashed. When looking at the mountpoint it was showing files with garbled names and messed up user/group owners.
mdadm --detail /dev/md5 showed that 3 disks were in a spare state? Like this: /dev/sdc1(S). Stopped and started the array, no go. Mdadm examine showed the disks were 'clean' so I force assembled and it did, just one disk missing. Kicked it out of the array, wiped superblock and put it back in, all looks ok now and it's rebuilding. Back on the old Sil3132 controller ofcourse.
So after having quite a scare my data might have been lost permantly;
- The asm1061 works fine on individual disks
- The asm1061 can see all disks behind the PM
- Performance on the disks behind the PM is fine
- When accessing all disks at once by generating IO to /dev/md5 it crawls to a halt
- The sil3132 works fine but just limited to 130MB/s
smartctl looks fine for each disk:
/dev/sdc
Code: |
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 174 174 021 Pre-fail Always - 6283
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 34
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 444
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 34
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 28
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 114
194 Temperature_Celsius 0x0022 117 115 000 Old_age Always - 33
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
|
/dev/sdd
Code: |
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 174 173 021 Pre-fail Always - 6300
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 12
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 204
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 7
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 52
194 Temperature_Celsius 0x0022 121 121 000 Old_age Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
|
/dev/sde
Code: |
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 177 177 021 Pre-fail Always - 6116
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 447
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 29
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 144
194 Temperature_Celsius 0x0022 120 118 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
|
/dev/sdf
Code: |
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 179 179 021 Pre-fail Always - 6008
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 447
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 29
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 116
194 Temperature_Celsius 0x0022 118 116 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
|
/dev/sdg
Code: |
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 182 181 021 Pre-fail Always - 5883
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 447
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 29
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 112
194 Temperature_Celsius 0x0022 116 114 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
|
Anyone any clue? hints? |
|
Back to top |
|
|
s4e8 Guru
Joined: 29 Jul 2006 Posts: 311
|
Posted: Thu Jan 24, 2013 9:11 am Post subject: |
|
|
asm chip don't support pmp fis base switch, it's non-capable for raid.
sil3132 has fbs support, but limited throughput per port.
I recommed buying a marvell 91xx base adapter. |
|
Back to top |
|
|
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Thu Jan 24, 2013 10:00 am Post subject: |
|
|
s4e8 wrote: | asm chip don't support pmp fis base switch, it's non-capable for raid.
sil3132 has fbs support, but limited throughput per port.
I recommed buying a marvell 91xx base adapter. |
Does it? On all spec pages it says that asm1061 is PMP compatible, also it can see all the drives behind the PMP and I have seen successes with drive cages on an asm1061. Would it not just only see 1 drive if it cannot handle FIS PMP?
Code: |
Engaged in High Speed I/O solution development, Asmedia Technology is committed to enlarging product portfolio with introducing PCI Express Products. The ASM1061, x1 PCI Express to two ports of Serial ATA, enables Serial ATA PHY up to 6Gbps high speed interface, following Serial ATA Revision 3.0 specification.
General Features
- Option Rom support through 64K Byte SPI flash
- 20MHz external crystal
- Integrated 3.3V to 1.2V switch regulator
- 3.3/1.2V Power Supply
- Industry Specifications Compliance:
PCI Express Base Specification Rev. 2.0
Bus Master Programming Interface to IDE ATA Controller Rev. 1.0
PCI Express Card Electromechanical Rev . 2.0
Serial ATA AHCI Spec. Rev.1.3
Serial ATA Revision 3.0
PCI Express Features
- x1 PCI Express lane for 2.5 and 5GHz signaling
- Single virtual channel
- SSC support
- ECRC and Advanced Error Reporting capability
- 100MHz differential PCI Express reference clock in
- Maximum Payload up to 128 bytes
Serial ATA Features
- 2 ports Serial ATA PHY for 1.5, 3.0 and 6.0GHz signaling
- Support Native Command Queue
- Support Gen 1m and 2m SATA PHY
- Support Port Multiplier |
I've looked around the web and found some other cards similar to this one that use the ASM1061 and they all state the controller is FIS-Based switching compatible.. Not found any whitepaper or datasheet from AsMedia yet.
This one for example: http://www.mediasonic.ca/product.php?id=1357626065
Code: |
ASMEDIA-asm1061 chipset
Support Port Multiplier and FIS-Based switch
|
|
|
Back to top |
|
|
s4e8 Guru
Joined: 29 Jul 2006 Posts: 311
|
Posted: Thu Jan 24, 2013 1:48 pm Post subject: |
|
|
ASM1061 has no FBS support, your dmesg say it.
After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.
Sil3132 is the cheapest working PMP adapter.
Another adapter has best sata pmp support is highpoint 272x SAS2 adapter(marvell 9485), w/ proprietary driver. The mvsas or other libsas based driver or all open source linux sas drivers has no sata pmp at all.
Add some info. After force kernel to ignore the fbs flag, Marvell 9172 work fine with the sata pmp. So the chip doesn't support fbs. |
|
Back to top |
|
|
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Mon Feb 04, 2013 12:32 pm Post subject: |
|
|
s4e8 wrote: | ASM1061 has no FBS support, your dmesg say it.
After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.
Sil3132 is the cheapest working PMP adapter.
Another adapter has best sata pmp support is highpoint 272x SAS2 adapter(marvell 9485), w/ proprietary driver. The mvsas or other libsas based driver or all open source linux sas drivers has no sata pmp at all.
Add some info. After force kernel to ignore the fbs flag, Marvell 9172 work fine with the sata pmp. So the chip doesn't support fbs. |
My understanding is that "FIS Based Switching" implies that the host controller is able to "tag" individual packets/blocks/actions (or whatever goes down a SATA wire) with the "label" of the destination disk. The port multiplier uses this "tag" to route the packet to the correct disk. Without this ability the host controller will only detect 1 disk, namely the one connected to the first most port of the multiplier. (As described in the manuals of the PM's)
Fact is that when the server boots the bios shows 5 connected drives, even better; booted into Gentoo I get 5 device nodes and I can benchmark every individual device. My conclusion would be that the controller supports FIS Based Switching. Else it would only 'see' disk number 1 behind the multiplier. It's only when you mount the MD array and generate traffic over all disk that things go belly up. I would say the kernel's ahci support for this controller is bugged/flaky rather then the controller not supporting FIS Based Switching and this needs patching.
Doesn't the fact that during boot 5 drives are visible and the fact that I can see and benchmark 5 different device nodes prove that the controller is capable of FBS?
I'm not arguing but looking for an explanation on why this isn't working. Any who can explain this behaviour to me? |
|
Back to top |
|
|
s4e8 Guru
Joined: 29 Jul 2006 Posts: 311
|
Posted: Mon Feb 04, 2013 2:18 pm Post subject: |
|
|
Command Based switch means You can switch disk by command, like flip papers. When you access disk1, you can't access disk2 concurrently. This mode just like the primary and slave disk on same PATA cable.
FIS is a request based switch, sth like the SCSI protocol. |
|
Back to top |
|
|
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Tue Feb 05, 2013 9:37 am Post subject: |
|
|
Ahhhh that makes sense. Thank you for this explanation.
It works fine when accessing individual disks and it detects all of them.. Benchmarking several makes it slow and when benchmarking the array things go bad. Probably MDADM doesn't like the delays and eventually fails a disk?
Bit disappointed as most websites advertise the controller as FIS capable. |
|
Back to top |
|
|
ciscoy2g15 n00b
Joined: 04 Dec 2015 Posts: 1
|
Posted: Fri Dec 04, 2015 2:35 pm Post subject: |
|
|
I know this thread is a bit old. But perhaps someone may find it useful. After having several RAID arrays get corrupted, and finally getting familiar with mdadm to be able to recover from these resets, I found this:
http://www.linuxquestions.org/questions/linux-hardware-18/sata-libata-link-reset-problem-with-port-multiplier-pmp-sil3132-sil3726-4175445070/
It looks like you can add the libata-pmp.ko kernel module so that you can prevent these resets. I haven't been successful in doing this for Ubuntu, and was hoping Gentoo already had it loaded by default, but wasn't the case. I guess I'll have to do a bit more reading about how to create my own kernel module based on this:
http://lxr.free-electrons.com/source/drivers/ata/libata-pmp.c
It seems that it's also dependent on a couple of other files, which you can find here (one directory up):
http://lxr.free-electrons.com/source/drivers/ata/
But kept getting problems when compiling the kernel module, stating that the symbols were incorrect or something to that effect (I'm away from that system, and don't want to bring it down to test at this time). At any rate, I just wanted to post this on here.
If anyone has any pointers as to how to go about compiling this kernel in either Gentoo or Ubuntu, it would be helpful. Otherwise, I'll post an update once I figure it out.
Peace |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|