| View previous topic :: View next topic |
| Author |
Message |
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
 Posted: Sun Mar 31, 2024 5:22 pm Post subject: Discard over USB |
 |
|
Hello,
this is the discussion thread about https://wiki.gentoo.org/wiki/Discard_over_USB ...
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
 |
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Sun Mar 31, 2024 5:54 pm Post subject: |
|
|
From this wiki I learned, that trim via USB bridges is not trivial.
I'm using btrfs on LUKS on all my devices that are HDDs, SATA SSDs and NVMe SSDs.
I came to investigate trim and discard further as sometimes by BTRFS is blocking all IO right after booting:
https://lore.kernel.org/linux-btrfs/238dc2b36f27838baf02425b364705c58fcc5de5.camel@gmx.net
The mount option discard maybe envolved in this issue so I disabled that on all devices...
What happens if an USB bridge does not support trim? Someday the SSD can't write anymore without moving to another controler to get the trim done?
I have a Ugreen SSD M.2 NVMe enclosure with USB 3.1 Gen2. Following the Wiki it first looks like supporting trim but then it fails:
| Code: | # lsusb.py |grep Ugreen
4-6 0bda:9210 00 1IF [USB 3.20, 5000 Mbps, 896mA] (Ugreen Ugreen Storage Device 0129380014F4)
# lsusb |grep 9210
Bus 004 Device 004: ID 0bda:9210 Realtek Semiconductor Corp. RTL9210 M.2 NVME Adapter
# eix -Ic usbutils
[I] sys-apps/usbutils (017@19.01.2024): USB enumeration utilities |
| Code: | # find /sys/ -name provisioning_mode -exec grep -H . {} + | sort
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-14/2-14:1.0/host6/target6:0:0/6:0:0:0/scsi_disk/6:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-14/2-14:1.0/host6/target6:0:0/6:0:0:1/scsi_disk/6:0:0:1/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-14/2-14:1.0/host6/target6:0:0/6:0:0:2/scsi_disk/6:0:0:2/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-14/2-14:1.0/host6/target6:0:0/6:0:0:3/scsi_disk/6:0:0:3/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-14/2-14:1.0/host6/target6:0:0/6:0:0:4/scsi_disk/6:0:0:4/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-7/2-7:1.0/host7/target7:0:0/7:0:0:0/scsi_disk/7:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb4/4-6/4-6:1.0/host5/target5:0:0/5:0:0:0/scsi_disk/5:0:0:0/provisioning_mode:unmap
/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:writesame_16
/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/scsi_disk/1:0:0:0/provisioning_mode:full
|
I guess this is my device:
| Code: | # find /sys/ -name provisioning_mode -exec grep -H . {} + | sort |grep 4-6
/sys/devices/pci0000:00/0000:00:14.0/usb4/4-6/4-6:1.0/host5/target5:0:0/5:0:0:0/scsi_disk/5:0:0:0/provisioning_mode:unmap |
| Code: | # fdisk -l /dev/sdh
Disk /dev/sdh: 3,64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: SN850X 4000GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: ...
Device Start End Sectors Size Type
/dev/sdh1 2048 73727 71680 35M EFI System
/dev/sdh2 73728 598015 524288 256M Linux filesystem
/dev/sdh3 598016 7814035455 7813437440 3,6T Linux filesystem |
| Code: | # sg_vpd -p bl /dev/sdh
Block limits VPD page (SBC):
Write same non-zero (WSNZ): 0
Maximum compare and write length: 0 blocks [Command not implemented]
Optimal transfer length granularity: 1 blocks
Maximum transfer length: 65535 blocks
Optimal transfer length: 65535 blocks
Maximum prefetch transfer length: 0 blocks [ignored]
Maximum unmap LBA count: 20971520
Maximum unmap block descriptor count: 1
Optimal unmap granularity: 1 blocks
Unmap granularity alignment valid: false
Unmap granularity alignment: 0 [invalid]
Maximum write same length: 0 blocks [not reported]
Maximum atomic transfer length: 0 blocks [not reported]
Atomic alignment: 0 [unaligned atomic writes permitted]
Atomic transfer length granularity: 0 [no granularity requirement
Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]
# sg_vpd -p bl /dev/sdh |grep "Maximum unmap LBA"
Maximum unmap LBA count: 20971520 |
Up to here, it looks like trim might be supported.
Unmap was already set because I ran the steps already before this post. Anyway, setting again:
| Code: | # echo unmap > /sys/devices/pci0000:00/0000:00:14.0/usb4/4-6/4-6:1.0/host5/target5:0:0/5:0:0:0/scsi_disk/5:0:0:0/provisioning_mode
# echo '20971520*512' | bc
10737418240
# echo $(echo '20971520*512' | bc) > /sys/block/sdh/queue/discard_max_bytes
-bash: echo: write error: Invalid argument
|
... and failing here. What can I do? What does this mean?
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55293
Location: 56N 3W
|
| Posted: Sun Mar 31, 2024 6:13 pm Post subject: |
|
|
Massimo B.
So far so good.
Is using | Code: | | echo 10737418240 > /sys/block/sdh/queue/discard_max_bytes |
any help?
If trim cannot be enabled, the drive will do some clean up, when a write is commanded. It will have to do an erase first.
Trim allows the drive to do the clean up in advance of the write that would force it, thus avoiding the slow erase cycle slowing writes.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Sun Mar 31, 2024 6:19 pm Post subject: |
|
|
Does that mean, if trim is not supported, that the device is getting slower when getting more bytes written over time, or that every write is just always slower on this USB bridge?
That echo is failing as mentioned above:
| Code: | # echo 10737418240 > /sys/block/sdh/queue/discard_max_bytes
-bash: echo: write error: Invalid argument |
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55293
Location: 56N 3W
|
| Posted: Sun Mar 31, 2024 6:29 pm Post subject: |
|
|
Massimo B.
What does | Code: | | cat /sys/block/sdh/queue/discard_max_hw_bytes |
tell?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Tue Apr 02, 2024 6:40 am Post subject: |
|
|
| Code: | # echo 10737418240 > /sys/block/sdh/queue/discard_max_bytes
-bash: echo: write error: Invalid argument
# cat /sys/block/sdh/queue/discard_max_bytes
4294966784 |
Does that discard_max_bytes need to know anything about the partitioning and the LUKS device, or does it only see the block device? Why is that set size smaller than expected?
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Thu Jun 27, 2024 11:52 am Post subject: |
|
|
Any idea how to enable discard on that USB-NVMe bridge?
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Mon Mar 31, 2025 8:27 pm Post subject: |
|
|
A new device, same question:# | Code: | # sg_vpd -p bl /dev/sda |grep "unmap LBA"
Maximum unmap LBA count: 0x1400000
# find /sys/ -name provisioning_mode -exec grep -H . {} + | sort
/sys/devices/pci0000:00/0000:00:0d.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:full
# sg_readcap -l /dev/sda | grep length
Logical block length=512 bytes |
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6288
Location: Dallas area
|
| Posted: Mon Mar 31, 2025 8:36 pm Post subject: |
|
|
| Code: | $ cat /etc/udev/rules.d/50-usb-ssd-trim.rules
ACTION=="add", ATTRS{idVendor}=="174c", ATTRS{idProduct}=="235c", SUBSYSTEM=="scsi_disk", RUN+="/bin/sh -c 'sleep 1 && udevadm trigger --subsystem-match=scsi_disk'"
ACTION=="change", ATTRS{idVendor}=="174c", ATTRS{idProduct}=="235c", SUBSYSTEM=="scsi_disk", ATTR{provisioning_mode}="unmap" |
use your vendor product id
_________________
UM780, 6.14 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Mon Mar 31, 2025 9:21 pm Post subject: |
|
|
I don't think that setting the unmap is a problem. I have already set it manually as described in the Wiki: | Code: | # find /sys/ -name provisioning_mode -exec grep -H . {} + | sort
/sys/devices/pci0000:00/0000:00:0d.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:unmap |
But doing a fstrim -av does not list the currently mounted usb device.
I fail at the step in the Wiki where the bc calculation for /sys/block/sda/queue/discard_max_bytes is done. How can I do that with my numbers?
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6288
Location: Dallas area
|
| Posted: Mon Mar 31, 2025 9:53 pm Post subject: |
|
|
I've never had to set discard_max.
But, if it doesn't show with the fstrim all command then the drive might not support trim.
from the man page
| Quote: | | Trim all mounted filesystems on devices that support the discard operation. |
What does "fstrim -v <mount point of disk>" do? Any messages?
And if no messages what does "fstrim -v <mnt>; echo $?" show?
_________________
UM780, 6.14 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Mon Mar 31, 2025 10:13 pm Post subject: |
|
|
| Code: | # fstrim -v /mnt/usb/mobiledata/
fstrim: /mnt/usb/mobiledata/: the discard operation is not supported
# echo $?
1 |
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6288
Location: Dallas area
|
| Posted: Mon Mar 31, 2025 10:47 pm Post subject: |
|
|
| Massimo B. wrote: | | Code: | # fstrim -v /mnt/usb/mobiledata/
fstrim: /mnt/usb/mobiledata/: the discard operation is not supported
# echo $?
1 |
|
So either the drive itself doesn't support discard OR the usb bridge doesn't pass it on.
_________________
UM780, 6.14 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Mon Mar 31, 2025 10:59 pm Post subject: |
|
|
At least they promise to support UASP and TRIM on that recent model: https://de.ugreen.com/collections/festplattengehause/products/35576
At least for UASP they are right. Trim I haven't seen working. Trim and discard is actually the same, is it?
For LUKS I enabled allow_discard: | Code: | # cryptsetup luksDump /dev/sda3 |grep Flags
Flags: allow-discards |
On top of that LUKS device I have btrfs: | Code: | # mount |grep mobile
/dev/mapper/mobiledata_crypt on /mnt/autofs/usb/mobiledata type btrfs (rw,relatime,compress-force=zstd:15,ssd,space_cache=v2,subvolid=5,subvol=/) |
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6288
Location: Dallas area
|
| Posted: Tue Apr 01, 2025 9:27 am Post subject: |
|
|
I have several ugreen ssd cases and they work well.
But I don't know how luks interacts with things like btrfs and fstrim.
Yes trim and discard are the same thing. Users typically say trim, while disk manf like to use discard.
Edit to add: googling fstrim on luks device leads to
| Quote: | | encryption (cryptsetup LUKS) prevents TRIM by default, needs "allow discards" flag. where to set this flag depends on where/what opens the crypt container. |
https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives
scroll down to "Discard/TRIM support for solid state drives"
_________________
UM780, 6.14 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1902
Location: PB, Germany
|
| Posted: Tue Apr 01, 2025 9:58 am Post subject: |
|
|
AFAIK, allowing discard happens in 2 levels, the first is on the LUKS device, doing a | Code: | | cryptsetup luksOpen --allow-discards --persistent ... |
The second is on filesystem mount adding the discard option.
I added the discard now to the btrfs mount, and now the fstrim -av is working on the USB but failing:
| Code: | # mount |grep mobile
/dev/mapper/mobiledata_crypt on /mnt/autofs/usb/mobiledata type btrfs (rw,relatime,compress-force=zstd:15,ssd,space_cache=v2,subvolid=5,subvol=/)
# fstrim -av
fstrim: /mnt/autofs/usb/mobiledata: the discard operation is not supported
/var/cache/fscache: 0 B (0 bytes) trimmed on /dev/loop0
/boot/efi: 11,8 MiB (12390400 bytes) trimmed on /dev/nvme0n1p1
/boot: 0 B (0 bytes) trimmed on /dev/nvme0n1p2
/: 8,8 GiB (9439563776 bytes) trimmed on /dev/mapper/luks-e226f36d-9aa2-1001-a2f0-b5370aaad063 |
I just wonder, the root filesystem had no discard set on the btrfs mount either but was still processed by fstrim -av: | Code: | | /dev/mapper/luks-e226f36d-9aa2-1001-a2f0-b5370aaad063 on / type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=282,subvol=/volumes/root) |
_________________
HP ZBook Power G9 i7-12700H|HP ProDesk 600 G5 i7-9700|HP EliteDesk 800 G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
sublogic
Guru
Joined: 21 Mar 2022
Posts: 334
Location: Pennsylvania, USA
|
| Posted: Wed Apr 02, 2025 1:47 am Post subject: |
|
|
| Massimo B. wrote: | AFAIK, allowing discard happens in 2 levels, the first is on the LUKS device, doing a | Code: | | cryptsetup luksOpen --allow-discards --persistent ... |
The second is on filesystem mount adding the discard option. |
Yes for the cryptsetup option, but beware of the mount option. https://wiki.gentoo.org/wiki/SSD#Mounting : | Quote: | | For rootfs it is usually recommended to periodically use fstrim utility. Using the discard mount option results in continuous discard that could potentially cause degradation of older or poor-quality SSDs |
(For what it's worth: I have a USB3 backup drive, a shingled HDD. I can't get fstrim to work on it either.) |
|
| Back to top |
|
|
|
Kernel & Hardware
