Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
AsMedia ASM1061 Port Multiplier issues
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 43

PostPosted: Thu Jan 24, 2013 7:43 am    Post subject: AsMedia ASM1061 Port Multiplier issues Reply with quote

Hi All,

I'm running an Intel D510MO mainboard with has a MiniPCIe slot. According to specs it has a PCIe 1x v2 bus capable of 500MB/s. Currently it has a Silicon Image 3132 based 2-port SATA controller installed, the second SATA port is attached to an Addonics SATA port multiplier. The PM is Sil3726 based. There are 5 3TB WD30EZRX disks attached to the PM stuffed in an mdadm raid5 array.

This works fine but I am not able to get more throughput than approx 130MB/s through the Sil3132. It appears that despite being SATA300 the card/chip has an internal limit that is bottlenecking my disk IO.

After doing some research I found that the Asmedia 1061 chip is very cheap and it is capable of reaching speeds well over 300MB/s. Because I'm running a SATA300 port multiplier I don't expect to make those speeds but at least well over 130MB/s would be nice, 250MB/s would be more in line with my expectation.

I went ahead and got myself one of these: http://www.delock.de/produkte/F_426_Module_95225/merkmale.html

2 port MiniPCIe based SATA controller that uses an Asmedia 1061 chip. Plugged it in and the BIOS shows the card is in AHCI mode, as I already had AHCI SATA enabled in the kernel it was picked up just fine:

Code:

[    0.622850] ahci 0000:02:00.0: irq 45 for MSI/MSI-X
[    0.622924] ahci: SSS flag set, parallel bus scan disabled
[    0.623107] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
[    0.623305] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part ccc sxs
[    0.624803] scsi4 : ahci
[    0.625285] scsi5 : ahci
[    0.625691] ata5: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200100 irq 45
[    0.625889] ata6: SATA max UDMA/133 abar m512@0xf0200000 port 0xf0200180 irq 45


It finds the disks through the port multiplier:

Code:

[    1.948274] ata5.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[    1.948438] ata5.01: hard resetting link
[    2.440248] ata5.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.440408] ata5.02: hard resetting link
[    2.932252] ata5.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    2.932410] ata5.03: hard resetting link
[    3.424248] ata5.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    3.424407] ata5.04: hard resetting link
[    3.916245] ata5.04: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    3.916403] ata5.05: hard resetting link
[    7.236028] ata5.15: qc timeout (cmd 0xe4)
[    7.236154] ata5.05: failed to read SCR 1 (Emask=0x4)
[    7.236269] ata5.05: failed to read SCR 0 (Emask=0x40)
[    7.237029] ata5.00: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[    7.237148] ata5.00: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    7.237910] ata5.00: configured for UDMA/133
[    7.238611] ata5.01: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[    7.238732] ata5.01: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    7.239484] ata5.01: configured for UDMA/133
[    7.240187] ata5.02: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[    7.240312] ata5.02: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    7.241070] ata5.02: configured for UDMA/133
[    7.241771] ata5.03: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[    7.241892] ata5.03: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    7.242648] ata5.03: configured for UDMA/133
[    7.243350] ata5.04: ATA-9: WDC WD30EZRX-00DC0B0, 80.00A80, max UDMA/133
[    7.243472] ata5.04: 5860533168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    7.244241] ata5.04: configured for UDMA/133
[    7.244429] ata5: EH complete


And after booting the array was properly assembled. "Benchmarking" individual disks looks fine:

Code:

/dev/sdc:
 Timing cached reads:   1574 MB in  2.00 seconds = 787.55 MB/sec
 Timing buffered disk reads: 438 MB in  3.01 seconds = 145.53 MB/sec

/dev/sdd:
 Timing cached reads:   1540 MB in  2.00 seconds = 770.47 MB/sec
 Timing buffered disk reads: 476 MB in  3.00 seconds = 158.51 MB/sec

/dev/sde:
 Timing cached reads:   1540 MB in  2.00 seconds = 770.61 MB/sec
 Timing buffered disk reads: 426 MB in  3.02 seconds = 141.08 MB/sec

/dev/sdf:
 Timing cached reads:   1544 MB in  2.00 seconds = 772.48 MB/sec
 Timing buffered disk reads: 424 MB in  3.02 seconds = 140.44 MB/sec

/dev/sdg:
 Timing cached reads:   1548 MB in  2.00 seconds = 773.76 MB/sec
 Timing buffered disk reads: 428 MB in  3.01 seconds = 142.16 MB/sec


The real problem occurs when accessing the array:

Code:

/dev/md5:
 Timing cached reads:   1428 MB in  2.00 seconds = 714.01 MB/sec
 Timing buffered disk reads:  70 MB in  3.34 seconds =  20.93 MB/sec


In comparison; the Sil3132 does this (while rebuilding actually):

Code:

/dev/md5:
 Timing cached reads:   1258 MB in  2.00 seconds = 629.35 MB/sec
 Timing buffered disk reads: 358 MB in  3.01 seconds = 118.80 MB/sec


Stumped as I was I ran the hdparm a couple of times to make sure the result was consistent. The results were even worse; sometimes 2MB/s up to 7MB/s. Hmm weird; so to confirm I wrote a 1GiB test file on the array which was very slow. After a minute or so it stalled and the array kinda crashed. When looking at the mountpoint it was showing files with garbled names and messed up user/group owners.

mdadm --detail /dev/md5 showed that 3 disks were in a spare state? Like this: /dev/sdc1(S). Stopped and started the array, no go. Mdadm examine showed the disks were 'clean' so I force assembled and it did, just one disk missing. Kicked it out of the array, wiped superblock and put it back in, all looks ok now and it's rebuilding. Back on the old Sil3132 controller ofcourse.

So after having quite a scare my data might have been lost permantly;

- The asm1061 works fine on individual disks
- The asm1061 can see all disks behind the PM
- Performance on the disks behind the PM is fine
- When accessing all disks at once by generating IO to /dev/md5 it crawls to a halt
- The sil3132 works fine but just limited to 130MB/s

smartctl looks fine for each disk:

/dev/sdc
Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   174   174   021    Pre-fail  Always       -       6283
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       34
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       444
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       34
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       28
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       114
194 Temperature_Celsius     0x0022   117   115   000    Old_age   Always       -       33
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


/dev/sdd
Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   174   173   021    Pre-fail  Always       -       6300
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       204
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       52
194 Temperature_Celsius     0x0022   121   121   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0


/dev/sde
Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   177   021    Pre-fail  Always       -       6116
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       144
194 Temperature_Celsius     0x0022   120   118   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


/dev/sdf
Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   179   179   021    Pre-fail  Always       -       6008
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       116
194 Temperature_Celsius     0x0022   118   116   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


/dev/sdg
Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   182   181   021    Pre-fail  Always       -       5883
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       447
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       29
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       112
194 Temperature_Celsius     0x0022   116   114   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


Anyone any clue? hints? :roll:
Back to top
View user's profile Send private message
s4e8
Apprentice
Apprentice


Joined: 29 Jul 2006
Posts: 205

PostPosted: Thu Jan 24, 2013 9:11 am    Post subject: Reply with quote

asm chip don't support pmp fis base switch, it's non-capable for raid.
sil3132 has fbs support, but limited throughput per port.
I recommed buying a marvell 91xx base adapter.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 43

PostPosted: Thu Jan 24, 2013 10:00 am    Post subject: Reply with quote

s4e8 wrote:
asm chip don't support pmp fis base switch, it's non-capable for raid.
sil3132 has fbs support, but limited throughput per port.
I recommed buying a marvell 91xx base adapter.


Does it? On all spec pages it says that asm1061 is PMP compatible, also it can see all the drives behind the PMP and I have seen successes with drive cages on an asm1061. Would it not just only see 1 drive if it cannot handle FIS PMP?

Code:

Engaged in High Speed I/O solution development, Asmedia Technology is committed to enlarging product portfolio with introducing PCI Express Products. The ASM1061, x1 PCI Express to two ports of Serial ATA, enables Serial ATA PHY up to 6Gbps high speed interface, following Serial ATA Revision 3.0 specification.

General Features
- Option Rom support through 64K Byte SPI flash
- 20MHz external crystal
- Integrated 3.3V to 1.2V switch regulator
- 3.3/1.2V Power Supply
- Industry Specifications Compliance:
   PCI Express Base Specification Rev. 2.0
   Bus Master Programming Interface to IDE ATA Controller Rev. 1.0
   PCI Express Card Electromechanical Rev . 2.0
   Serial ATA AHCI Spec. Rev.1.3
   Serial ATA Revision 3.0
 
PCI Express Features
- x1 PCI Express lane for 2.5 and 5GHz signaling
- Single virtual channel
- SSC support
- ECRC and Advanced Error Reporting capability
- 100MHz differential PCI Express reference clock in
- Maximum Payload up to 128 bytes

Serial ATA Features
- 2 ports Serial ATA PHY for 1.5, 3.0 and 6.0GHz signaling
- Support Native Command Queue
- Support Gen 1m and 2m SATA PHY
- Support Port Multiplier


I've looked around the web and found some other cards similar to this one that use the ASM1061 and they all state the controller is FIS-Based switching compatible.. Not found any whitepaper or datasheet from AsMedia yet.

This one for example: http://www.mediasonic.ca/product.php?id=1357626065

Code:

ASMEDIA-asm1061 chipset
Support Port Multiplier and FIS-Based switch
Back to top
View user's profile Send private message
s4e8
Apprentice
Apprentice


Joined: 29 Jul 2006
Posts: 205

PostPosted: Thu Jan 24, 2013 1:48 pm    Post subject: Reply with quote

ASM1061 has no FBS support, your dmesg say it.
After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.
Sil3132 is the cheapest working PMP adapter.
Another adapter has best sata pmp support is highpoint 272x SAS2 adapter(marvell 9485), w/ proprietary driver. The mvsas or other libsas based driver or all open source linux sas drivers has no sata pmp at all.

Add some info. After force kernel to ignore the fbs flag, Marvell 9172 work fine with the sata pmp. So the chip doesn't support fbs.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 43

PostPosted: Mon Feb 04, 2013 12:32 pm    Post subject: Reply with quote

s4e8 wrote:
ASM1061 has no FBS support, your dmesg say it.
After do some testing with Marvell 9172. The linux ahci driver don't detect disks at all. The windows does see disks but the performance is poor.
Sil3132 is the cheapest working PMP adapter.
Another adapter has best sata pmp support is highpoint 272x SAS2 adapter(marvell 9485), w/ proprietary driver. The mvsas or other libsas based driver or all open source linux sas drivers has no sata pmp at all.

Add some info. After force kernel to ignore the fbs flag, Marvell 9172 work fine with the sata pmp. So the chip doesn't support fbs.


My understanding is that "FIS Based Switching" implies that the host controller is able to "tag" individual packets/blocks/actions (or whatever goes down a SATA wire) with the "label" of the destination disk. The port multiplier uses this "tag" to route the packet to the correct disk. Without this ability the host controller will only detect 1 disk, namely the one connected to the first most port of the multiplier. (As described in the manuals of the PM's)

Fact is that when the server boots the bios shows 5 connected drives, even better; booted into Gentoo I get 5 device nodes and I can benchmark every individual device. My conclusion would be that the controller supports FIS Based Switching. Else it would only 'see' disk number 1 behind the multiplier. It's only when you mount the MD array and generate traffic over all disk that things go belly up. I would say the kernel's ahci support for this controller is bugged/flaky rather then the controller not supporting FIS Based Switching and this needs patching.

Doesn't the fact that during boot 5 drives are visible and the fact that I can see and benchmark 5 different device nodes prove that the controller is capable of FBS?

I'm not arguing but looking for an explanation on why this isn't working. Any who can explain this behaviour to me? :?:
Back to top
View user's profile Send private message
s4e8
Apprentice
Apprentice


Joined: 29 Jul 2006
Posts: 205

PostPosted: Mon Feb 04, 2013 2:18 pm    Post subject: Reply with quote

Command Based switch means You can switch disk by command, like flip papers. When you access disk1, you can't access disk2 concurrently. This mode just like the primary and slave disk on same PATA cable.
FIS is a request based switch, sth like the SCSI protocol.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 43

PostPosted: Tue Feb 05, 2013 9:37 am    Post subject: Reply with quote

Ahhhh that makes sense. Thank you for this explanation.

It works fine when accessing individual disks and it detects all of them.. Benchmarking several makes it slow and when benchmarking the array things go bad. Probably MDADM doesn't like the delays and eventually fails a disk?

Bit disappointed as most websites advertise the controller as FIS capable. :evil:
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum