USB-sticks... playing with them... and having errors

[Zucca]

Soo... Not so serious probleam here, but I bought two 128GB Verbatim USB sticks for one of my boxes.
I've used those sticks a lot before at my work. So this time I might have just bad luck.

Firstly lsblk shows this about them:

Code: Select all

sdc                     8:32   1 115.2G  0 disk
sdd                     8:48   1 115.4G  0 disk

... which is odd... The other one has less capacity.
I planned to use them as a distfiles storage so I created lvm vg and one raid0 lv (--stripes 2). Then formatted the lv with xfs.
Problems began:

Code: Select all

M600-ctrl ~ # mkfs.xfs -L pkg -m reflink=1 /dev/tUSB/pkg 
meta-data=/dev/tUSB/pkg          isize=512    agcount=16, agsize=524272 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=0   metadir=0
data     =                       bsize=4096   blocks=8388352, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=0
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
mkfs.xfs: pwrite failed: I/O error
libxfs_bwrite: write failed on xfs_agf bno 0x37ff901/0x1, err=5

and these messages appeared at dmesg:

Code: Select all

[136511.110033] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136541.833411] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136579.379982] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136616.926713] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136654.473400] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136692.019948] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136698.313765] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=218s
[136698.313803] sd 2:0:0:0: [sdc] tag#0 Sense Key : Unit Attention [current] 
[136698.313828] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Not ready to ready change, medium may have changed
[136698.313850] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 01 c0 a4 81 00 00 01 00
[136698.313864] I/O error, dev sdc, sector 29402241 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
[136698.314027] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[136698.314057] I/O error, dev sdc, sector 241663872 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[136729.563167] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136760.286781] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136797.843396] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136835.379984] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136872.926556] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136910.469853] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[136916.768375] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=218s
[136916.768411] sd 2:0:0:0: [sdc] tag#0 Sense Key : Unit Attention [current] 
[136916.768435] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Not ready to ready change, medium may have changed
[136916.768457] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 01 c0 a4 83 00 00 01 00
[136916.768472] I/O error, dev sdc, sector 29402243 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2

Note the blocksize xfs chose is 4k. But the stripe size for the raid0 is 64k. This shouldn't matter, right?

As those are USB sticks, I doubt any smart cheks are valid. Just in case I used shred -vzn 2 on both of them. No errors or lines printed on dmesg. Then I proceeded to read all the data (zeroes) from both - no errors, again. So random data writing and zeroing works and reading zeros work.

I also partitioned both sticks afterwards and left 20MB gaps at both ends and created the space for pv at the middle. Same errors.

I've changed the USB ports too.
My guess is that sdc has some sectors broken.

I take any wild guesses about what's going on. :P Just in case someone else has had this situation.
Thanks.

EDIT: s/Trancend/Verbatim/

[Zucca]

The errors indicate it the stick with slightly less space every time...

Code: Select all

[167213.509125] sd 2:0:0:0: [sdc] tag#0 Sense Key : Unit Attention [current] 
[167213.509149] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Not ready to ready change, medium may have changed
[167213.509172] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 00 0e a8 00 00 00 08 00
[167213.509187] I/O error, dev sdc, sector 960512 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[167213.509234] Buffer I/O error on dev dm-5, logical block 229376, lost async page write
[167213.509389] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167213.509422] I/O error, dev sdc, sector 241663872 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[167244.758160] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167275.491307] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167313.028068] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167350.574764] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167388.121396] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167425.668060] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167431.962840] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=218s
[167431.962876] sd 2:0:0:0: [sdc] tag#0 Sense Key : Unit Attention [current] 
[167431.962902] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Not ready to ready change, medium may have changed
[167431.962924] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 00 0e a8 08 00 00 20 00
[167431.962939] I/O error, dev sdc, sector 960520 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2
[167431.962984] Buffer I/O error on dev dm-5, logical block 229377, lost async page write
[167431.963040] Buffer I/O error on dev dm-5, logical block 229378, lost async page write
[167431.963075] Buffer I/O error on dev dm-5, logical block 229379, lost async page write
[167431.963105] Buffer I/O error on dev dm-5, logical block 229380, lost async page write
[167431.966253] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.966281] I/O error, dev sdc, sector 1222664 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 2
[167431.966318] Buffer I/O error on dev dm-5, logical block 294913, lost async page write
[167431.966351] Buffer I/O error on dev dm-5, logical block 294914, lost async page write
[167431.966382] Buffer I/O error on dev dm-5, logical block 294915, lost async page write
[167431.966408] Buffer I/O error on dev dm-5, logical block 294916, lost async page write
[167431.966602] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.966625] I/O error, dev sdc, sector 2140160 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[167431.966659] Buffer I/O error on dev dm-5, logical block 524288, lost async page write
[167431.966837] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.966857] I/O error, dev sdc, sector 2140232 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[167431.966886] Buffer I/O error on dev dm-5, logical block 524297, lost async page write
[167431.968651] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.968659] I/O error, dev sdc, sector 2140248 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[167431.969193] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.969200] I/O error, dev sdc, sector 1222656 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2
[167431.969344] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.969350] I/O error, dev sdc, sector 3319808 op 0x1:(WRITE) flags 0x800 phys_seg 5 prio class 2
[167431.969380] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167431.969385] I/O error, dev sdc, sector 3581952 op 0x1:(WRITE) flags 0x800 phys_seg 5 prio class 2
[167463.211544] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167493.934833] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167531.481355] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167569.028005] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167606.574848] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167644.118078] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167650.416244] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=218s
[167650.416285] sd 2:0:0:0: [sdc] tag#0 Sense Key : Unit Attention [current] 
[167650.416309] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Not ready to ready change, medium may have changed
[167650.416331] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 00 40 a8 00 00 00 08 00
[167650.416347] I/O error, dev sdc, sector 4237312 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[167650.416393] buffer_io_error: 12 callbacks suppressed
[167650.416404] Buffer I/O error on dev dm-5, logical block 1048576, lost async page write
[167650.423392] sd 2:0:0:0: [sdc] tag#0 device offline or changed
[167650.423410] I/O error, dev sdc, sector 6334464 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 2
[167650.423434] Buffer I/O error on dev dm-5, logical block 1572864, lost async page write
[167650.423454] Buffer I/O error on dev dm-5, logical block 1572865, lost async page write
[167681.664550] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd
[167712.388049] usb 1-5.1: reset high-speed USB device number 6 using xhci_hcd

... there I tried to format the raid0'd LV to ext4.

I'll setup the bigger stick and leave the other for testings.

[Zucca]

Ok now the other, previously working one, died too. It's now RO. I tried just to format it as ext4.
I'll return both sticks and try something else.

Any suggestions for reliable USB sticks? :D USB 2.0 is fine.

[zen_desu]

Where did you buy them? I wouldn't expect "real" transcend brand flash to be that sketchy.

I generally get sandisk USBs, I don't think I've had one fail me, even with my habit of clearing them with random data before/after I use them again.


Side note, if doing USB raid, I'd try to put each device on a different USB root.

[szatox]

Any suggestions for reliable USB sticks? USB 2.0 is fine.

No idea whether this is a tip you need, but I always look for the long ones, with full metal body and no moving parts. I want this thing to have as much cooling as it can get*, and hopefully a physically bigger chip inside with more and bigger transistors rather than abusing smaller cells with more bits stuffed at tighter tolerances.

Don't know about brands, just look at the product pictures in whatever online store / price search you use. Bonus points for specifying actual memory speed, since most only list USB speed I could just as well get from wikipedia, thank you very much.


* for durability AND write speeds, the hotter they get, the slower they run. Occasionally, I've been even cooling them with wet tissues. It does help with heavy load.

[Zucca]

Where did you buy them? I wouldn't expect "real" transcend brand flash to be that sketchy.

I bought them from the store where I work. ;) I have used the same series of USB sticks quite a lot, but the capacity of 16Gb and 32Gb ones. None of them have failed.
And I have to correct myself as those are Verbatim branded 2.0 sticks.

I generally get sandisk USBs, I don't think I've had one fail me, even with my habit of clearing them with random data before/after I use them again.

Ok that sounds good. Thanks.

Side note, if doing USB raid, I'd try to put each device on a different USB root.

For bandwidth reasons, I assume? Yeah. I probably could just go with some kind of JBOD.

The point of this storage setup is to have some local storage for non-essential files (distfiles mainly). So I'll be using at least noatime too.

No idea whether this is a tip you need, but I always look for the long ones, with full metal body and no moving parts. I want this thing to have as much cooling as it can get*

This is why I plug these sticks into USB 2.0 ports, so that the bandwidth is limited and they can't get too hot. ;)

[NeddySeagoon]

Zucca,

Use a reputable brand that you buy at a reasonable price.
Prices that look too good to be true usually are.
It's like the old saying about oats.
If you want good quality oats you must pay a fair price. If you want oats that have already been through the horse ... :)

I use Sandisk and overprovision quite a lot. I usually get sticks at the 'sweet spot' in terms of GB/<currency>.

Don't use a jouronalling fs unless you really need it.
Few if any USB sticks support trim/discard so the write speed falls off lots when space is being overwitten. You need to wait for erase cycles.
Ext4 is fine with ^has_journal.
Does distfiles need to be recovered or can you throw it away.

Raid over USB. Don't do that. USB is too flakey.

[Zucca]

Zucca,

Use a reputable brand that you buy at a reasonable price.
Prices that look too good to be true usually are.

I relied on Verbatim. And also though the price was low because USB 2.0 interface and the speed of the flash chips.

It's like the old saying about oats.
If you want good quality oats you must pay a fair price. If you want oats that have already been through the horse ... :)

... that might have happened here. ;)

I use Sandisk and overprovision quite a lot.

How do you do it? Just leave empty space when partitioning? Or do you use lvm to limit how much of space can be used on a PV? Flash controller is "in a way" of custom overprovision usually... partially.

I usually get sticks at the 'sweet spot' in terms of GB/<currency>.

Don't use a jouronalling fs unless you really need it.
Few if any USB sticks support trim/discard so the write speed falls off lots when space is being overwitten. You need to wait for erase cycles.
Ext4 is fine with ^has_journal.
Does distfiles need to be recovered or can you throw it away.

I guess f2fs should be a good fit for storage such as "usb-stick-distfiles". With some tuning like noatime etc.

Raid over USB. Don't do that. USB is too flakey.

Not even a simple raid such as raid1 or raid0?

[NeddySeagoon]

Zucca,

Raid over USB3, when you have UAS may be OK. I've not tried it.
With raid over USB2, I've tried a few times, many years ago, and gave up.

For over provisioning,I just leave unused space on FLASH devices.
Keep in mind that with FLASH storage, the concept of a partition being a contiguous sequence of blocks is a myth, perpetuated by FLASH memory controllers.
Much like the concept of CHS addressing of hard drives has been since the introduction of LBA and zoned drives.

If you have a need for speed, why start with USB2?

[Goverp]

With some tuning like noatime etc.

AFAIK these days kernel defaults already include atime optimizations to almost eliminate IO for datestamps. See "relatime" and "lazyatime".

[C5ace]

I had a dud Verbatim 16GB USB Stick with metal cover. The stick was not visible with fdisk. Only with lsblk. Solution was run dd if=/dev/zero of=/dev/sdf bs=512 count=(as shown by lsblk) status=progress. No error shown. Then used fdisk to create a ext4 partition. Formatted the partition with ext4.. The USB stick the auto mounted when plugged in.

The first attempt with bs=1M failed.

[Spanik]

For such semi-permanent flash storage I use a sata to SD card board. You can find them with 1-2-4 cards. Most do the raid for you but you do not have much control over it. Depends on your use case I guess.