Disk layout suggestions for NVMe SSD

[bytex]

Hello, I’m explaining my goals. I’d appreciate it if experienced members could leave their comments. Thank you.
I have a WD SN7100 1TB Gen4 NVMe SSD and I’m preparing the partitioning scheme.
Here is current plan:

nvme0n1p1 → EFI System Partition 512mb or 1gb
nvme0n1p2 → LVM and LUKS rest 931.51 Mib

Inside the LVM setup:
Separate LV + FS for / (Once I wanted to use F2FS, but I didn’t dare and went with ext4)
Separate LV + FS for /home (When root was ext4, I had made this one F2FS on the same setup)
Separate LV + FS for /opt (and I set this up with XFS)
Separate LV + FS for /var (XFS)
Separate LV + FS for /srv (XFS)
/tmp → mounted as RAM tmpfs
swap → to swap

My first Redcore Linux layout was like above
But Yesterday, I accidentally wrote to the entire disk by testing with fio, and after the reboot whole installation was gone.
Now I decided to install plain Gentoo from scratch second time.

Actually installed Gentoo from scratch for the first time, but it really wore me out, I got stuck and it was exhausting.
Especially During the kernel compilation step there were too many options, and since I got tired of waiting, I had installed Redcore instead.

but this low speed issue still bothers me. On this nvme, the raw 4K random read Q1T1 speed is 276 MB/s, but when I tested on a partition like ext4 I got 104 MB/s in some test 50 mb/s
I have no idea as im new to this Nvme thing.. I couldn’t achieve the fast and snappy system I was hoping for
I bought this WD SN7100 1TB specifically because it has one of the highest 4K random Q1T1 speeds, thinking my Linux system disk would feel snappier.

I want to keep exactly the same layout as above, use ext4, f2fs, and xfs filesystems all together.*
Still, I couldn’t reach the performance I wanted from this NVMe. I checked on my motherboard and confirmed that it’s running at Gen4 speed. Also, I haven’t tested it on very old kernels.
The UEFI power settings are also set to maximum.

If a professional user had this disk, how would they install Gentoo on it as their main system? What NVMe tweaks and adjustments would they apply?

[C5ace]

ext4 on SAMSUNG MZAL8512HDLU-00BL2
/etc/fstab:

Code: Select all

 
# <fs>              <mountpoint>    <type>	<opts>     <dump/pass>
LABEL=L1_UEFI       /efi            vfat	umask=0077	0 2
LABEL=L1_ROOT       /               ext4	noatime		0 1
LABEL=L1_SWAP       none            swap	sw		     0 0
LABEL=L1_HOME	    /home           ext4	noatime      0 2

Copy speed test:

Code: Select all

# ~/Desktop $ /var/db/repos # time cp -R gentoo gentoo-2

real	0m3.996s
user	0m0.243s
sys	0m3.686s

[pingtoo]

but this low speed issue still bothers me. On this nvme, the raw 4K random read Q1T1 speed is 276 MB/s, but when I tested on a partition like ext4 I got 104 MB/s in some test 50 mb/s

Can you share what tool you used to get "the raw 4K random read Q1T1 speed is 276 MB/s" number? and what tool you used to get "tested on a partition like ext4 I got 104 MB/s " number?

The reason I ask is because it sound strange to me, if same tool used in both test, there is no reason for the difference because partition layout.

However if they are from different tools and/or different Operating System than it could happen but that comparison I would say it is meaningless because how the tool implement read/write operation matter.

[NeddySeagoon]

bytex,

Read the fine print in your motherboard manual, or post the motherboard link, so we can see for ourselves.

Some low end motherboards ho not have enough PCIe lanes to go around, so they are shared.
That means that they are taken away from from things like NVMe slots to use for SATA.

That is, its a design feature. If you want to use all for PCIe lanes for NVMe, you cannot use SATA at the same time.
Motherboards vary.

Speed test benchmarks, without knowing the tool in use, don't say very much.

[bytex]

but this low speed issue still bothers me. On this nvme, the raw 4K random read Q1T1 speed is 276 MB/s, but when I tested on a partition like ext4 I got 104 MB/s in some test 50 mb/s

Can you share what tool you used to get "the raw 4K random read Q1T1 speed is 276 MB/s" number? and what tool you used to get "tested on a partition like ext4 I got 104 MB/s " number?

The reason I ask is because it sound strange to me, if same tool used in both test, there is no reason for the difference because partition layout.

However if they are from different tools and/or different Operating System than it could happen but that comparison I would say it is meaningless because how the tool implement read/write operation matter.

Yeah.
250 mb/s chimera@redcore ~ $ sudo fio --name=rand4ktest --filename=/dev/nvme0n1 --rw=randrw --bs=4k --size=1G --numjobs=4 --time_based --runtime=60s --ioengine=libaio --direct=1 --group_reporting: https://pastebin.pl/view/5a30f6aa

Sorry, I couldn’t find the rest atm, but I’ll search again. 104 MB/s is also reported by the KdiskMark (The Linux equivalent of CrystalDiskMark) for 4K random read Q1T1.

bytex,

Read the fine print in your motherboard manual, or post the motherboard link, so we can see for ourselves.

Some low end motherboards ho not have enough PCIe lanes to go around, so they are shared.
That means that they are taken away from from things like NVMe slots to use for SATA.

That is, its a design feature. If you want to use all for PCIe lanes for NVMe, you cannot use SATA at the same time.
Motherboards vary.

Speed test benchmarks, without knowing the tool in use, don't say very much.

Hmm, I hadn’t thought of that, maybe! this is my motherboard it's kinda new: https://en.colorful.cn/en/home/product? ... 3088e5d34c”
Yes, right now I have two disks connected via SATA, one is a SATA SSD and the other is a HDD. Do you think this was the problem? Using all three at the same time?

[NeddySeagoon]

bytex,

What does

Code: Select all

 --direct=1 

do?
If its what I think it is, it prevents the use of kernel and HDD speedups by forcing direct IO. Thus commands are not queued, buffered or anything else.
Once command must complete before the next starts.
That's not how it works in reality and gives low speed test results. Low compared to everyday use that is,

Its still a valid speed test to compare different devices.

as your motherboard is PCIe 5 its unlikely to need to share PCIe lanes.