| View previous topic :: View next topic |
| Author |
Message |
mijenix
Guru
Joined: 22 Apr 2003
Posts: 393
Location: Switzerland
|
 Posted: Fri Nov 26, 2010 10:59 pm Post subject: raid1: Disk failure - smartctl can't find anything, fault? |
 |
|
Hi
Today my raid1 had again the following problem (the error occurred 2 times until now):
| Quote: |
Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953187152
Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953187400
Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953186896
Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953187144
Nov 19 22:04:16 reese kernel: raid1: Disk failure on sdc1, disabling device.
Nov 19 22:04:16 reese kernel: raid1: Operation continuing on 1 devices.
Nov 19 22:04:16 reese kernel: raid1: sdb1: redirecting sector 1953187152 to another mirror
Nov 19 22:04:16 reese kernel: raid1: sdb1: redirecting sector 1953187400 to another mirror
Nov 19 22:04:16 reese kernel: raid1: sdb1: redirecting sector 1953186896 to another mirror
Nov 19 22:04:16 reese kernel: raid1: sdb1: redirecting sector 1953187144 to another mirror
|
now it only runs with one device.
I ran smartctl -t short and -t long but it can't find any error, following the smartctl output:
| Code: |
smartctl --all /dev/sdc
smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green family
Device Model: WDC WD10EADS-00L5B1
Serial Number: WD-WCAU46928032
Firmware Version: 01.01A01
User Capacity: 1,000,204,886,016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Fri Nov 26 23:45:45 2010 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (22200) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 163 162 021 Pre-fail Always - 6816
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 304
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 4358
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 302
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 4
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 304
194 Temperature_Celsius 0x0022 115 108 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 4358 -
# 2 Short offline Completed without error 00% 4354 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
|
The first time the error occured I re-added the disk to the raid and it worked for 2 months.
But I'm a little bit worried, any ideas what the problem can be, what do you think?
Thanks in advice. |
|
| Back to top |
|
 |
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sat Nov 27, 2010 12:59 am Post subject: |
|
|
| Smart is not everything... try a long self test, also try accessing these sectors directly using dd seek, see if it works or not. If it does work, maybe you have an erratic cable or controller. Otherwise, the disk is just plain bad and smart is not smart enough to know about it. |
|
| Back to top |
|
|
mijenix
Guru
Joined: 22 Apr 2003
Posts: 393
Location: Switzerland
|
| Posted: Sat Nov 27, 2010 10:54 am Post subject: |
|
|
I re-added the device to the raid and got this
| Code: |
ov 27 10:24:25 reese kernel: md: bind<sdc1>
Nov 27 10:24:25 reese kernel: RAID1 conf printout:
Nov 27 10:24:25 reese kernel: --- wd:1 rd:2
Nov 27 10:24:25 reese kernel: disk 0, wo:0, o:1, dev:sdb1
Nov 27 10:24:25 reese kernel: disk 1, wo:1, o:1, dev:sdc1
Nov 27 10:24:25 reese kernel: md: recovery of RAID array md1
Nov 27 10:24:25 reese kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Nov 27 10:24:25 reese kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Nov 27 10:24:25 reese kernel: md: using 128k window, over a total of 976759936 blocks.
Nov 27 10:24:25 reese mdadm[1988]: RebuildStarted event detected on md device /dev/md1
Nov 27 10:29:48 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7fffffff SErr 0xc80900 action 0x6 frozen
Nov 27 10:29:48 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:00:bf:e2:3e/00:00:03:00:00/40 tag 0 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:08:3f:e3:3e/00:00:03:00:00/40 tag 1 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:10:bf:e4:3e/00:00:03:00:00/40 tag 2 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:18:3f:e5:3e/00:00:03:00:00/40 tag 3 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:20:bf:e5:3e/00:00:03:00:00/40 tag 4 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:28:3f:e6:3e/00:00:03:00:00/40 tag 5 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:30:bf:e6:3e/00:00:03:00:00/40 tag 6 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:38:3f:e7:3e/00:00:03:00:00/40 tag 7 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:40:bf:e7:3e/00:00:03:00:00/40 tag 8 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:48:3f:e8:3e/00:00:03:00:00/40 tag 9 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:50:bf:e8:3e/00:00:03:00:00/40 tag 10 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:58:3f:e9:3e/00:00:03:00:00/40 tag 11 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:60:bf:e9:3e/00:00:03:00:00/40 tag 12 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:68:3f:ea:3e/00:00:03:00:00/40 tag 13 ncq 65536 out
Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:70:bf:ea:3e/00:00:03:00:00/40 tag 14 ncq 65536 out
...
ov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:29:48 reese kernel: ata6: hard resetting link
Nov 27 10:29:49 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov 27 10:29:49 reese kernel: ata6.00: configured for UDMA/133
Nov 27 10:29:49 reese kernel: ata6: EH complete
Nov 27 10:30:01 reese cron[3898]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
Nov 27 10:35:46 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7ffeffff SErr 0xc80900 action 0x6 frozen
Nov 27 10:35:46 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error
Nov 27 10:35:46 reese kernel: ata6.00: cmd 61/80:00:3f:62:dd/00:00:06:00:00/40 tag 0 ncq 65536 out
Nov 27 10:35:46 reese kernel: res 40/00:7c:bf:63:dd/00:00:06:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:35:46 reese kernel: ata6.00: cmd 61/80:08:bf:62:dd/00:00:06:00:00/40 tag 1 ncq 65536 out
...
Nov 27 10:35:46 reese kernel: res 40/00:7c:bf:63:dd/00:00:06:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:35:46 reese kernel: ata6: hard resetting link
Nov 27 10:35:47 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov 27 10:35:47 reese kernel: ata6.00: configured for UDMA/133
Nov 27 10:35:47 reese kernel: ata6: EH complete
Nov 27 10:36:14 reese kernel: ata6.00: exception Emask 0x50 SAct 0x1ffff SErr 0xc80900 action 0x6 frozen
Nov 27 10:36:14 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error
...
Nov 27 10:36:14 reese kernel: res 40/00:24:3f:1f:25/00:00:07:00:00/40 Emask 0x50 (ATA bus error)
Nov 27 10:36:14 reese kernel: ata6: hard resetting link
Nov 27 10:36:15 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov 27 10:36:15 reese kernel: ata6.00: configured for UDMA/133
Nov 27 10:36:15 reese kernel: ata6: EH complete
Nov 27 10:37:52 reese kernel: ata6: limiting SATA link speed to 1.5 Gbps
Nov 27 10:37:52 reese kernel: ata6.00: exception Emask 0x10 SAct 0x7fffbfff SErr 0x480100 action 0x6 frozen
Nov 27 10:37:52 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error
Nov 27 10:37:52 reese kernel: ata6.00: cmd 61/80:00:3f:ef:1f/00:00:08:00:00/40 tag 0 ncq 65536 out
...
Nov 27 10:37:54 reese kernel: res 40/00:3c:bf:36:20/00:00:08:00:00/40 Emask 0x52 (ATA bus error)
Nov 27 10:37:54 reese kernel: ata6: hard resetting link
Nov 27 10:37:54 reese kernel: ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Nov 27 10:37:54 reese kernel: ata6.00: failed to IDENTIFY (I/O error, err_mask=0x100)
Nov 27 10:37:54 reese kernel: ata6.00: revalidation failed (errno=-5)
Nov 27 10:37:59 reese kernel: ata6: hard resetting link
Nov 27 10:38:00 reese kernel: ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Nov 27 10:38:00 reese kernel: ata6.00: configured for UDMA/133
Nov 27 10:38:00 reese kernel: ata6: EH complete
Nov 27 10:38:00 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7ffffff SErr 0xc80900 action 0x6 frozen
Nov 27 10:38:00 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error
Nov 27 10:38:00 reese kernel: ata6.00: cmd 61/80:00:3f:4d:20/00:00:08:00:00/40 tag 0 ncq 65536 out
...
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Unhandled error code
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] CDB: cdb[0]=0x2a: 2a 00 08 21 72 3f 00 01 00 00
Nov 27 10:38:28 reese kernel: end_request: I/O error, dev sdc, sector 136409663
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Unhandled error code
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] CDB: cdb[0]=0x2a: 2a 00 08 21 71 3f 00 00 80 00
Nov 27 10:38:28 reese kernel: end_request: I/O error, dev sdc, sector 136409407
...
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] READ CAPACITY(16) failed
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Sense not available.
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] READ CAPACITY failed
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Sense not available.
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Asking for cache data failed
Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Assuming drive cache: write through
Nov 27 10:38:28 reese kernel: sdc: detected capacity change from 1000204886016 to 0
Nov 27 10:38:29 reese kernel: RAID1 conf printout:
Nov 27 10:38:29 reese kernel: --- wd:1 rd:2
Nov 27 10:38:29 reese kernel: disk 0, wo:0, o:1, dev:sdb1
Nov 27 10:38:29 reese kernel: disk 1, wo:1, o:0, dev:sdc1
Nov 27 10:38:29 reese kernel: RAID1 conf printout:
Nov 27 10:38:29 reese kernel: --- wd:1 rd:2
Nov 27 10:38:29 reese kernel: disk 0, wo:0, o:1, dev:sdb1
Nov 27 10:38:29 reese mdadm[1988]: RebuildFinished event detected on md device /dev/md1
Nov 27 10:38:29 reese mdadm[1988]: FailSpare event detected on md device /dev/md1, component device /dev/sdc1
|
I plugged the sata-cable out and in and started the server, now the raid is rebuilding.
When the error occurs again I'll change the faulty disk.
Do you think the controller can be faulty?
On that controller the system disk (sda) and the non faulty raid disk (sdb) is attached. Those disks had not any problems until now. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55146
Location: 56N 3W
|
| Posted: Sat Nov 27, 2010 11:11 am Post subject: |
|
|
mijenix,
If the fault has cleared, its probably a problem with poor quality connectors on the data cable. It will be OK for about a year, then it will happen again - due to the poor quality connectors.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
mijenix
Guru
Joined: 22 Apr 2003
Posts: 393
Location: Switzerland
|
| Posted: Sat Nov 27, 2010 3:08 pm Post subject: |
|
|
so after the resync the raid1 seems fine.
I know the cable problem from my eSATA backupdrive.
The cable hasn't any plug fixation but the external disk is more affected due to agitation then the internal.
Are there any better sata cables then the mobo provided ones?
Thanks again for all your help. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55146
Location: 56N 3W
|
| Posted: Sat Nov 27, 2010 4:57 pm Post subject: |
|
|
mijenix,
Almost any SATA cables are better than motherboard provided ones.
SATA cables are a commodity item. Look at cheap ones, look at expensive ones, then buy in the middle of the price range, or just a little above.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
