[solved] I/O errors on disk

[lordalbert]

Hi,
i've some problems of I/O Errors. When i perform some operations on an ssd disk, it gives me errors. For example, when i format a partition (mkfs.ext4 command), it complete the job but it warns me with errors.
Now i launch "fstrim -a" to disk and it stopped with I/O errors.
After those errors, often (not always) all system broke down. Every command gives me "I/O error". if i do a "ls", it gives me errors. Opened programs crashed and can't open again.
The only think i can do in this situation is to reboot the system and all return to normal.

I fear that i have a problem with my ssd health, so i installed vendor tool on win (Crucial Storage Executive) to perform a "long test" and it passed. I checked SMART on that tool and it returns a "good status". I noticed there was a firmware upgrade (a security fix), i do it but nothing changed. So it seems that the SSD status is good...

Do you know what could be the problem and how to solve it?

I attach a smart output of the disk

Code: Select all

$ sudo smartctl -a /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.80-gentoo-r1] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron BX/MX1/2/3/500, M5/600, 1100 SSDs
Device Model:     Crucial_CT250MX200SSD1
Serial Number:    1614124C4694
LU WWN Device Id: 5 00a075 1124c4694
Firmware Version: MU05
User Capacity:    250.059.350.016 bytes [250 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jan 27 11:38:33 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(  795) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (   5) minutes.
Conveyance self-test routine
recommended polling time: 	 (   3) minutes.
SCT capabilities: 	       (0x0035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocate_NAND_Blk_Cnt 0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       3460
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       589
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 Ave_Block-Erase_Count   0x0032   099   099   000    Old_age   Always       -       31
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       122
180 Unused_Reserve_NAND_Blk 0x0033   000   000   000    Pre-fail  Always       -       2591
183 SATA_Interfac_Downshift 0x0032   100   100   000    Old_age   Always       -       23
184 Error_Correction_Count  0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   062   047   000    Old_age   Always       -       38 (Min/Max 15/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       6
202 Percent_Lifetime_Remain 0x0030   099   099   001    Old_age   Offline      -       1
206 Write_Error_Rate        0x000e   100   100   000    Old_age   Always       -       0
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000    Old_age   Always       -       0
246 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       4457025264
247 Host_Program_Page_Count 0x0032   100   100   000    Old_age   Always       -       139371123
248 FTL_Program_Page_Count  0x0032   100   100   000    Old_age   Always       -       144226845

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      3459         -
# 2  Short offline       Completed without error       00%      3457         -
# 3  Short offline       Aborted by host               00%      3457         -
# 4  Extended offline    Completed without error       00%      3431         -
# 5  Vendor (0xff)       Completed without error       00%      3419         -
# 6  Vendor (0xff)       Completed without error       00%      3250         -
# 7  Vendor (0xff)       Completed without error       00%      3130         -
# 8  Vendor (0xff)       Completed without error       00%      2729         -
# 9  Vendor (0xff)       Completed without error       00%      2640         -
#10  Vendor (0xff)       Completed without error       00%      2533         -
#11  Vendor (0xff)       Completed without error       00%      2499         -
#12  Vendor (0xff)       Completed without error       00%      2444         -
#13  Vendor (0xff)       Completed without error       00%      2367         -
#14  Vendor (0xff)       Completed without error       00%      2276         -
#15  Vendor (0xff)       Completed without error       00%      1970         -
#16  Vendor (0xff)       Completed without error       00%      1743         -
#17  Vendor (0xff)       Completed without error       00%      1631         -
#18  Vendor (0xff)       Completed without error       00%      1596         -
#19  Vendor (0xff)       Completed without error       00%      1571         -
#20  Vendor (0xff)       Completed without error       00%      1456         -
#21  Vendor (0xff)       Completed without error       00%      1372         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

thank you very much

[NeddySeagoon]

lordalbert,

The long test reads every block on the drive and as you say, it passed.
All your SMART data looks good too, except

Code: Select all

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
183 SATA_Interfac_Downshift 0x0032   100   100   000    Old_age   Always       -       23 

That implies you have interface errors that were recovered by running the SATA interface at 3G or even 1.5G.
Short term, disconnecting the SATA data cable both ends and reconnecting it may fix it.
It can also be the SATA port on the motherboard failing. Try another one.

If it is the cable, replace it with a mid price one. You get what you pay for and cheap SATA data cables are to be avoided.

When you get disc I/O errors, Linux makes the affected filesystem(s) read only to limit damage.
Almost nothing works after that.

[lordalbert]

Thank you.
I tried to change cable, but it seems that i resolved only changing sata port on the motherboard. So, probably a motherboard port failed... it is new so i'll claim the warranty

[NeddySeagoon]

lordalbert,

That sounds good.

[figueroa]

Thank you.
I tried to change cable, but it seems that i resolved only changing sata port on the motherboard. So, probably a motherboard port failed... it is new so i'll claim the warranty

To confirm failing port, try swapping it back. Just as likely that you corrected a poor connection.

[lordalbert]

Thank you.
I tried to change cable, but it seems that i resolved only changing sata port on the motherboard. So, probably a motherboard port failed... it is new so i'll claim the warranty

To confirm failing port, try swapping it back. Just as likely that you corrected a poor connection.

i tried. I still have errors on formatting a partition. I paste output (some text are in italian, sorry, but i higlight the error line:

Code: Select all

Discarding device blocks: failed - Errore di I/O remoto

The full output:

Code: Select all

# mkfs.ext4 /dev/sda5 -L vm_storage
mke2fs 1.45.6 (20-Mar-2020)
/dev/sda5 contiene un file system ext4 con etichetta "vm_storage"
	created on Wed Jan 27 22:12:21 2021
Proceed anyway? (y,N) y
Discarding device blocks: failed - Errore di I/O remoto
Creazione del file system con 31319854 4k blocchi e 7831552 inode
Etichetta del file system=8587d96e-e5e2-4f23-833a-598fb61fd546
Backup del superblocco salvati nei blocchi: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: fatto                           
Scrittura delle tavole degli inode: fatto                           
Creating journal (131072 blocks): fatto
Scrittura delle informazioni dei super-blocchi e dell'accounting del file system: fatto

but this time "fstrim -a" gives no error. It is better.

Now i'd like to test the sata port of motherboard, to understand if it is really failed. Do you know if there is a tool for that type of testing?

[NeddySeagoon]

lordalbert,

Code: Select all

Discarding device blocks: failed - Errore di I/O remoto 

That means that the discard command failed.

When you make an ext4 filesystem on an SSD, mkfs.ext4 tells the drive that the content og than region of the drive is no longer required.

If the SATA port was in IDE oc compatibility mode that might be expected.

What does lspci have to say about your SATA ports?

[lordalbert]

lordalbert,

Code: Select all

Discarding device blocks: failed - Errore di I/O remoto 

That means that the discard command failed.

When you make an ext4 filesystem on an SSD, mkfs.ext4 tells the drive that the content og than region of the drive is no longer required.

If the SATA port was in IDE oc compatibility mode that might be expected.

What does lspci have to say about your SATA ports?

So in that case it could be expected that also "fstrim" fails?

this is lscpi output related to SATA

Code: Select all

01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

[NeddySeagoon]

lordalbert,

Code: Select all

AHCI mode

Is the right answer. However,

Code: Select all

01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 

The mode is not shown.

You need to poke about in the firmware to ensure the legacy/IDE mode is not in use for those two chipsets.
If its not an option, its not in use. It was only ever intended for one time use by Windows users so that they could install the AHCI driver, then switch away from legacy/IDE mode forever. Due to its intended one time use, the mode is often only partially functional.

[lordalbert]

lordalbert,

Code: Select all

AHCI mode

Is the right answer. However,

Code: Select all

01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 

The mode is not shown.

You need to poke about in the firmware to ensure the legacy/IDE mode is not in use for those two chipsets.
If its not an option, its not in use. It was only ever intended for one time use by Windows users so that they could install the AHCI driver, then switch away from legacy/IDE mode forever. Due to its intended one time use, the mode is often only partially functional.

In firmware i haven't the legacy/ide choice (confirmed alsa by manual)! I can choose only between AHCI and RAID, and i have only 1 possibile choice (i think for all port), not different choices for different ports.

[NeddySeagoon]

lordalbert,

:)