[solved] Massive slowdowns when handling many small files

[Jimini]

Dear all,

when handling many small files with gentoo-sources-6.1.*, everything works fine.
With gentoo-sources-6.6.*, the transfer gets slower after a while. A reboot helps for a few moments, until the performance problem occurs again. I have to add, that a normal reboot does not work then, since one of the processes cannot be stopped or killed. Thus, I have to hard reset the system, e.g. with a magic SysRQ.

It does not make any difference how the files are transferred - for testing purposes, I transfer the extracted kernel sources (/usr/src/linux/) from A to B:
- local rsync from SSD to HDD RAID6
- rsync from a NFS share to a local folder
- cp within the same RAID6
- rsync within the same RAID6
- extracting a tarball on the RAID6
For now, a file transfer within the same local SSD seems to work fine.

I have tested a few kernel versions, with the following results:
gentoo-sources-6.1.19 -> OK
gentoo-sources-6.1.28 -> OK
gentoo-sources-6.1.67 -> OK
gentoo-sources-6.6.13 -> problems
gentoo-sources-6.6.62 -> problems
(I did not configure and compile every new kernel from scratch. For new kernel versions, I reuse the old config and, after a "make syncconfig", I compile the new kernel.)

The CPU load looks fine, so does the memory consumption.I assume, that some I/O stuff may be the root cause here, but I honestly have no idea how to verify this.

Since I have a workaround by simply using an older kernel, it does not look like a hardware (memory, CPU, RAID6) problem to me.

I would be thankful for any input for further testing.

Kind regards
Jimini

[pietinger]

[...](I did not configure and compile every new kernel from scratch. For new kernel versions, I reuse the old config and, after a "make syncconfig", I compile the new kernel.) [...]

Dont use "make syncconfig" ... use "make oldconfig". See also:
https://wiki.gentoo.org/wiki/User:Pieti ... fig.22_.3F

[pingtoo]

So is only destination RAID6 show this symptom? does reverse destination to SSD show same symptom?

It is worthy to know it is write bound or read bound.

[wjb]

Code: Select all

$ grep -a3 syncconfig /usr/src/linux/scripts/kconfig/Makefile
...
#
# Note:
#  syncconfig has become an internal implementation detail and is now
#  deprecated for external use

[Jimini]

pietinger, pingtoo & wjb, thank you for your replies.

Regarding syncconfig, it seems like I did not get the memo ;)
I have adapted my scripts and rebuilt the kernel on this particular machine, thank you.

I ran a few tests:
- unpacking a tarball on the SSD took 1m43s
- unpacking the same tarball on the RAID6 was canceled by me after there was no progress for 75min
- unpacking the same tarball from the RAID6 to the SSD took 1min10s
- unpacking the same tarball from the SSD to the RAID6 was canceled by me after 38mins

Writing to the RAID6 seems to lead to problems.
This is not the case in general - I can copy a 25G file to the RAID6, which takes about 7min (about 60MB/s).

In addition, I had a look at the processes on the system, while I was trying to reboot. "kworker/u8:0+flush-253:0" was at about 100% CPU load.

Maybe I should add share some information on my setup:
- Intel Core i3-9100 CPU @ 3.60GHz, 4 GB memory
- software RAID6, consisting of 7 HDDs (Seagate Exos X16, Toshiba MG08ACA16TE)
- on it is a LUKS2 container with aes-xts-plain64 as cipher
- ext4 is the filesystem:
dumpe2fs 1.47.1 (20-May-2024)
Filesystem volume name: <none>
Last mounted on: /home/share
Filesystem UUID: 94c5bf33-25cb-4eb0-9081-6e638dfabe67
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
[...]

Please bear in mind that with gentoo-sources-6.1.*, it works without problems :)
Any idea what I could test yet?

Kind regards
Jimini

[pietinger]

I could take a look at your kernel configuration ... preferably together with all three data mentioned here:
https://wiki.gentoo.org/wiki/User:Pieti ... nformation

[Jimini]

pietinger, thanks a lot for your offer.

Here you are:
emerge --info: https://paste.gentoo.zip/BhPUxQ13
dmesg: https://paste.gentoo.zip/0e1IKBml
lspci -nnk: https://paste.gentoo.zip/fZ9hw8oR (the command throws "lspci: Unable to load libkmod resources: error -2", but I guess this should not be the root cause for my problem)
kernel config: https://paste.gentoo.zip/q2mJlIVk

Again, I just tried to unpack the kernel files right on the RAID6, just for having a look at the dmesg output. But the only new line is the following:
[ 525.705770] EXT4-fs (dm-0): mounted filesystem 94c5bf33-25cb-4eb0-9081-6e638dfabe67 r/w with ordered data mode. Quota mode: none.


From today on, I will be on a trip until next sunday, so I will not be able to respond until then. So please take your time for having a look into the shared information.

Best regards and thanks in advance
Jimini

[pietinger]

Jimini,

you have as profile selected: default/linux/amd64/23.0/split-usr/no-multilib/hardened
... but you dont have a hardened kernel, you know that?

Your dmesg is not complete: I am missing the beginning; it starts at timestamp [ 1.477459]

I see you have not much RAM: 3.685.320 byte

It is an (old) Intel machine: Intel-R-_Core-TM-_i3-9100 and CONFIG_MCORE2=y

You have disabled module support (like me) and get the harmless error from lspci (I also have) because the package sys-apps/pciutils (which contains lspci) was emerged with Use-flag "kmod". If you want to get rid of the error message, you have to switch off this use flag for this package (but as I said, this is absolutely harmless).

I am missing some important options in your kernel, e.g. CONFIG_X86_X2APIC=y because you miss CONFIG_IRQ_REMAP ... because you switched it off completely:

Code: Select all

# CONFIG_INTEL_IOMMU is not set

(see more here: https://wiki.gentoo.org/wiki/User:Pieti ... rnel#IOMMU )

Maybe you will need also:

Code: Select all

# CONFIG_X86_INTEL_LPSS is not set
... AND one of them:
# CONFIG_LPC_ICH is not set
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_INTEL_LPSS_ACPI is not set
# CONFIG_MFD_INTEL_LPSS_PCI is not set

(see more here: https://wiki.gentoo.org/wiki/User:Pieti ... rm_support )

I always recommend to boot with our Gentoo LiveCD and do a "lsmod" and notice all loaded modules. Maybe you see then:

Code: Select all

# CONFIG_INTEL_IDMA64 is not set

is loaded instead of

Code: Select all

CONFIG_INTEL_IOATDMA=y

Some additional notes:

Code: Select all

1.
CONFIG_PREEMPT_NONE=y
2.
# CONFIG_SCHED_CORE is not set
3.
CONFIG_TRACEPOINTS=y
4.
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
3.
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
5.
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
6.
# CONFIG_INTEL_IDLE is not set
7.
CONFIG_IA32_EMULATION=y
8.
CONFIG_BLK_DEV_THROTTLING=y
3.
CONFIG_BLK_DEBUG_FS=y
9.
CONFIG_EXTRA_FIRMWARE=""
10.
CONFIG_I2C_ALI1535=y
CONFIG_I2C_ALI1563=y
CONFIG_I2C_ALI15X3=y
CONFIG_I2C_AMD756=y
CONFIG_I2C_AMD756_S4882=y
CONFIG_I2C_AMD8111=y
CONFIG_I2C_I801=y
CONFIG_I2C_PIIX4=y
CONFIG_I2C_NFORCE2=y
CONFIG_I2C_SIS5595=y
CONFIG_I2C_SIS630=y
CONFIG_I2C_SIS96X=y
CONFIG_I2C_VIA=y
CONFIG_I2C_VIAPRO=y

CONFIG_I2C_OCORES=y
CONFIG_I2C_SIMTEC=y
CONFIG_I2C_TAOS_EVM=y
CONFIG_I2C_TINY_USB=y

# CONFIG_PINCTRL is not set
11.
CONFIG_SECURITY_SELINUX=y
CONFIG_INTEGRITY=y
3.
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_FS_ALLOW_ALL=y
CONFIG_BLK_DEV_IO_TRACE=y

1. Tells me it was configured as server machine - Okay.
2. I recommend to enable it
3. Every debugging and tracing cost performance. I always recommend to switch it off (and use it only when investigating a specific problem)
4. Maybe you need it (the default value is: enabled)
5. I recommend to enable it
6. I recommend to enable it
7. You have a No-multilib system (like me) and can/should disable it
8. Really ? I would disable it.
9. I see you are using an initramfs (probably even built by yourself) - are you sure that the microcode and all firmware files are included? If not, include them in your kernel directly (https://wiki.gentoo.org/wiki/User:Pieti ... _Microcode ).
10. I think you will need only CONFIG_I2C_I801=y ... if at all ... because i2c requires pinctrl, which you have not activated. Do you have i2c hardware in this machine?
11. If you dont use it ... disable it.

[Jimini]

Good morning altogether,

first of all, thanks a lot @pietinger for your helpful input. Unfortunately, the system had some hardware issues and I had to replace the core components. Since then, I could not reproduce the problem anymore - but I will test it from time to time.

I will definitely have a look and your hints regarding the kernel config! Some of my configs are more than 15 years old, it is definitely possible that there is come old "misconfig heritage" ;)

Best regards
Jimini

[pietinger]

first of all, thanks a lot @pietinger for your helpful input. [...]

You are very Welcome!

[Ralphred]

Some of my configs are more than 15 years old, it is definitely possible that there is come old "misconfig heritage"

I know that feeling, if pietinger.net offered a "drop your lspci output and kernel .config here for expert feedback, only €5.99 a year" I'd subscribe

[pietinger]

I know that feeling, if pietinger.net offered a "drop your lspci output and kernel .config here for expert feedback, only €5.99 a year" I'd subscribe

Thanks a lot, Ralphred ... but I do it without costs ... especially for people who translate a german text into english (I dont have forgot that)

[Jimini]

Good morning,

as written last week, I have moved the system to newer hardware (AMD Ryzen 3 PRO 4350G, 16G RAM).

pietinger, I have now reconfigured my kernel (6.6.74) according to your hints - except for the Intel CPU related options, of course.
EXTRA_FIRMWARE is set to "amd-ucode/microcode_amd_fam17h.bin" (according to https://wiki.gentoo.org/wiki/Ryzen#Firmware).

I have now run a few tests again. Hereby I unpacked a tar.gz archive of /usr/src/linux/ - once on the SSD and once on the RAID6.
SSD: 0m13.652s
RAID6: 0m15.606s

Looks good so far, so I repeated the test:
SSD: 0m11.276s
RAID6: the process slows down drastically after a few seconds. While unpacking the archive on the RAID6 takes about one second for one single file to unpack, the result on the SSD is unchanged with 11.282s.

Looking into "top", the output is interesting:

Code: Select all

top - 11:17:38 up 6 min,  2 users,  load average: 2.16, 1.52, 0.69
Tasks: 180 total,   2 running, 178 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 11.9 sy,  0.0 ni, 75.0 id, 13.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15373.5 total,   8416.2 free,   1284.8 used,   5847.0 buff/cache
MiB Swap:   6144.0 total,   6144.0 free,      0.0 used.  14088.7 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  124 root      20   0       0      0      0 R 100.0   0.0   4:23.74 kworker/u+
    1 root      20   0    2472   1536   1536 S   0.0   0.0   0:01.00 init
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
[...]
 2277 root      20   0    5876   2560   2432 D   0.0   0.0   0:00.00 tar

Conclusions:
- We obviously have no CPU bottleneck here.
- According to "iotop", the RAID6 is accessed only by the tar process, at a few kilobytes per second.
- We have a "kworker" process, which fully loads one CPU core.
- On the old hardware, I could only reproduce the problem only on a kernel above version 6.1. With a 6.1 kernel, the problem did not occur. On this hardware, I have no bootable 6.1 kernel at hand, maybe I could test this later on.

dmesg: https://paste.gentoo.zip/MINvlkpL
emerge --info: https://paste.gentoo.zip/uVonOhJD
lspci -nnk: https://paste.gentoo.zip/Ke01mXdR
kernel config: https://paste.gentoo.zip/OdLLzFfD

I just stumbled upon viewtopic-p-8812952.html?sid=36f52eac3e ... fe02a0b89c - but the mentioned VM settings seem not to have any effect on my system, the problem persists.

...any ideas, what else to try?

Best regards
Jimini

[pietinger]

[...] ...any ideas, what else to try?

Hmm ... frankly, I don't know what the problem is ... if all the following doesn't help, then my last idea would be to use a different kernel version (it doesn't have to be the old 6.1. but take 6.12 - because that will soon be stable for Gentoo anyway) ... I would like to mention the following:

1) I'm sure you're aware of this yourself:

Code: Select all

[   10.742056] Warning: unable to open an initial console.
<=>
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1636] (rev da)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1636]
<=>
# CONFIG_DRM_AMDGPU is not set

=> https://wiki.gentoo.org/wiki/User:Pieti ... s_Firmware (because you have a monolithic kernel) => https://wiki.gentoo.org/wiki/AMDGPU

2) I guess this is/was the result of an hard reset:

Code: Select all

[   12.327137] EXT4-fs (sdh2): INFO: recovery required on readonly filesystem
[   13.732730] md/raid:md0: not clean -- starting background reconstruction

3) Please try to enable these both:

Code: Select all

# CONFIG_CHR_DEV_SG is not set
# CONFIG_SCSI_CONSTANTS is not set

(This is not relevant for your problem but should be enabled also: # CONFIG_I2C_PIIX4 is not set)

4) Do you run this kernel on bare metal or in a VM? If it is a VM then enable:

Code: Select all

# CONFIG_UDMABUF is not set

5) This is a shoot in the dark ... change it from 0 to 1:

Code: Select all

CONFIG_SATA_MOBILE_LPM_POLICY=0

Internal note:

Code: Select all

[    0.000000] Linux version 6.6.74-gentoo (root@share2) (gcc (Gentoo Hardened 14.2.1_p20241221 p7) 14.2.1 20241221, GNU ld (Gentoo 2.43 p3) 2.43.1) #3 SMP PREEMPT_DYNAMIC Sun Feb 16 10:04:59 CET 2025
[    1.901510] Memory: 15729892K/16121688K available (16384K kernel code, 847K rwdata, 3828K rodata, 11804K init, 2124K bss, 391536K reserved, 0K cma-reserved)
[    2.100062] smpboot: CPU0: AMD Ryzen 3 PRO 4350G with Radeon Graphics (family: 0x17, model: 0x60, stepping: 0x1)
[    2.107025] smp: Brought up 1 node, 8 CPUs

[    2.885892] smapi::smapi_init, ERROR invalid usSmapiID
[    2.885959] mwave: tp3780i::tp3780I_InitializeBoardData: Error: SMAPI is not available on this machine
[    2.886051] mwave: mwavedd::mwave_init: Error: Failed to initialize board data
[    2.886127] mwave: mwavedd::mwave_init: Error: Failed to initialize

[    2.886557] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

[    2.893957] megasas: 07.725.01.00-rc1
[    2.894039] mpt3sas version 43.100.00.00 loaded
[    2.894293] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (15730200 kB)

[    3.019930] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.020039] mpt2sas_cm0: MSI-X vectors supported: 1
[    3.020116] 	 no of cores: 8, max_msix_vectors: -1
[    3.020193] mpt2sas_cm0:  0 1 1
[    3.020353] mpt2sas_cm0: High IOPs queues : disabled
[    3.020431] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 32
[    3.020510] mpt2sas_cm0: iomem(0x00000000fcf40000), mapped(0x000000006fcf1c8f), size(65536)
[    3.020611] mpt2sas_cm0: ioport(0x000000000000f000), size(256)
[    3.075080] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.102826] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[    3.103033] mpt2sas_cm0: request pool(0x00000000e7d4f7f1) - dma(0x103300000): depth(3492), frame_size(128), pool_size(436 kB)
[    3.108544] mpt2sas_cm0: sense pool(0x00000000f8b86087) - dma(0x103a80000): depth(3367), element_size(96), pool_size (315 kB)
[    3.108721] mpt2sas_cm0: reply pool(0x0000000058ad46ef) - dma(0x103b00000): depth(3556), frame_size(128), pool_size(444 kB)
[    3.108827] mpt2sas_cm0: config page(0x000000004fb93642) - dma(0x103a40000): size(512)
[    3.108908] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[    3.108979] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[    3.109068] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    3.153821] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03)
[    3.153919] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    3.154131] scsi host0: Fusion MPT SAS Host
[    3.154765] mpt2sas_cm0: sending port enable !!

[   10.730207] pktgen: Packet Generator for packet performance testing. Version: 2.75

[   10.730525] GACT probability NOT on
[   10.730598] Mirror/redirect action on

01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
	Subsystem: Dell 6Gbps SAS HBA Adapter [1028:1f1c]
	Kernel driver in use: mpt3sas

CPU microcode update is correct/working:

Code: Select all

[   10.733824] microcode: microcode updated early to new patch_level=0x0860010d
[   10.733930] microcode: CPU2: patch_level=0x0860010d
[   10.733931] microcode: CPU0: patch_level=0x0860010d
...

[Jimini]

1) I'm sure you're aware of this yourself:

Code: Select all

[   10.742056] Warning: unable to open an initial console.
<=>
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1636] (rev da)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1636]
<=>
# CONFIG_DRM_AMDGPU is not set

=> https://wiki.gentoo.org/wiki/User:Pieti ... s_Firmware (because you have a monolithic kernel) => https://wiki.gentoo.org/wiki/AMDGPU

Yeah, but this should not be a problem here, since the system is headless :)

2) I guess this is/was the result of an hard reset:

Code: Select all

[   12.327137] EXT4-fs (sdh2): INFO: recovery required on readonly filesystem
[   13.732730] md/raid:md0: not clean -- starting background reconstruction

Exactly - when I reproduce the problem, I have a kworker process which prevents the system from shutting down. I then need to force a reboot, e.g. with a sysrq (sh -c "echo b > /proc/sysrq-trigger").

3) Please try to enable these both:

Code: Select all

# CONFIG_CHR_DEV_SG is not set
# CONFIG_SCSI_CONSTANTS is not set

(This is not relevant for your problem but should be enabled also: # CONFIG_I2C_PIIX4 is not set)

Done - unfortunately without effect.

4) Do you run this kernel on bare metal or in a VM? If it is a VM then enable:

Code: Select all

# CONFIG_UDMABUF is not set

The system is running on bare metal.

5) This is a shoot in the dark ... change it from 0 to 1:

Code: Select all

CONFIG_SATA_MOBILE_LPM_POLICY=0

Done - unfortunately without effect.




I ran my tests again, just for comparison I took the 6.1.127 kernel. I used my current config from 6.6.74 and adapted it with "make oldconfig". Then I changed my little script a bit - now it deletes the files after they have been decompressed.
Here are the results:

6.1.127:

Code: Select all

test 1 - decompressing tar.gz on SSD:

real    0m11.678s
user    0m8.851s
sys     0m8.750s

test 2 - decompressing tar.gz on RAID6

real    2m54.919s
user    0m11.804s
sys     0m12.612s

test 1 - decompressing tar.gz on SSD:

real    0m9.814s
user    0m8.406s
sys     0m6.630s

test 2 - decompressing tar.gz on RAID6

real    0m9.547s
user    0m8.637s
sys     0m6.358s

test 1 - decompressing tar.gz on SSD:

real    0m9.641s
user    0m8.519s
sys     0m6.517s

test 2 - decompressing tar.gz on RAID6

real    0m9.412s
user    0m8.563s
sys     0m6.311s

6.6.74:

Code: Select all

test 1 - decompressing tar.gz on SSD:

real    0m12.832s
user    0m8.466s
sys     0m5.534s

test 2 - decompressing tar.gz on RAID6

real    0m10.045s
user    0m8.492s
sys     0m5.479s

test 1 - decompressing tar.gz on SSD:

real    0m9.347s
user    0m8.394s
sys     0m5.407s

test 2 - decompressing tar.gz on RAID6

real    0m13.264s
user    0m8.365s
sys     0m5.512s

test 1 - decompressing tar.gz on SSD:

real    0m9.314s
user    0m8.519s
sys     0m5.315s

test 2 - decompressing tar.gz on RAID6

real    0m9.812s
user    0m8.445s
sys     0m5.316s

As you can see, I could not reproduce the problem anymore. On both kernels, the decompression of the archive worked without problems. In total, I ran 10 tests on the 6.6.74 kernel and the problem did not occur anymore.
The only difference now was, that while testing before, the target directory was overwritten. Now the extracted directory was deleted, so the process had to write all files every time.

So I let my script overwrite the target directory again:
- It does not matter which kernel I boot - just decompressing and writing the files into an empty directory works fine.
- When the target directory with all of its contents already exists, it looks like following:
6.1.127 & SSD: fast as expected
6.1.127 & RAID6: takes a lot longer (5mins and more)
6.6.74 & SSD: fast as expected
6.6.74 & RAID6: takes forever (I interrupted the task after ~40mins)

I cannot tell, where the difference between "write data to an empty dir" and "overwrite existing data" exactly is, but perhaps I am now one little step closer to the root cause...

Kind regards
Jimini

[pietinger]

[...] I ran my tests again, just for comparison [...]
I cannot tell, where the difference between "write data to an empty dir" and "overwrite existing data" exactly is, but perhaps I am now one little step closer to the root cause...

I would be very interested to see what happens with 6.12.14 (*). Only one test would be necessary - this one:

6.6.74 & RAID6: takes forever (I interrupted the task after ~40mins)

(Of course with the same config as before; updated with “make oldconfig”; answers to the questions are here: https://wiki.gentoo.org/wiki/User:Pieti ... perimental )

*) If ... IF ... there is no problem with 6.12 ... then I would think there is a kernel regression in 6.6


P.S.: If you test 6.12 please dont forget to update also package "linux-headers"

[Jimini]

Good morning,

thank you for your reply. I have now updated the sources and the headers (thanks for the hint, I would have forgotten that one!):
=sys-kernel/gentoo-sources-6.12.14
=sys-kernel/linux-headers-6.12

Then I copied the .config from the 6.6.74 kernel to the 6.12.14 directory, ran "make oldconfig" and compiled and installed the new kernel.
Thus, my system is now running on 6.12.14 (config: https://paste.gentoo.zip/Hamylfeu).

Code: Select all

kernel:
6.12.14-gentoo
test 2 - decompressing tar.gz on RAID6
real    0m11.202s
user    0m8.643s
sys     0m7.729s

test 2 - decompressing tar.gz on RAID6
real    0m14.094s
user    0m8.692s
sys     0m7.817s

test 2 - decompressing tar.gz on RAID6
real    0m19.357s
user    0m8.637s
sys     0m7.783s

test 2 - decompressing tar.gz on RAID6
real    0m18.047s
user    0m8.358s
sys     0m8.072s

Just for traceability, here is the little test script:
#!/bin/bash
echo "kernel:"
uname -r
echo "test 1 - decompressing tar.gz on SSD:"
cd /usr/src/test/
time tar -xzpf linux.tar.gz
#rm -rf /usr/src/test/linux-6.6.74-gentoo/
echo ""
echo "test 2 - decompressing tar.gz on RAID6"
cd /home/share/test/
time tar -xzpf linux.tar.gz
#rm -r /home/share/test/linux-6.6.74-gentoo

I also ran a backup script, which rsyncs a huge directory from my NFS server. This implies many directories with many small files, which could not be processed with a 6.6 kernel due to the slowdowns.
This script is now doing its job, like with kernel 6.1.

Seems like it is working again. Is there anything else I can test? What can we do now, since 6.6 seems to have a problem? :)

Kind regards
Jimini

[dmpogo]

It is getting interesting. I may note that 6.1 results are not clear either - you have odd 3 minutes in your first set of tests, and then 5 min
But if 6.12 is olso misbehaving, maybe one could bisect the commits between 6.1 and 6.6 that are related to SATA. I somewhow feel it is not software RAID per se.

[NeddySeagoon]

Jimini,

A few random thoughts ...

Code: Select all

[   13.732730] md/raid:md0: not clean -- starting background reconstruction

You can't do any useful speed tests with that going on.
Hopefully, you have a write intent bitmap so the reconstruction time is minimised?

Code: Select all

$ cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md127 : active raid5 sdc3[0] sda3[2] sdd3[3] sdb3[1]
      23441117184 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/59 pages [8KB], 65536KB chunk

Notice the 'bitmap' entry. It's not always useful, but every little helps. You have space for it in the mdadm header, so it costs nothing to add.

To flush your caches, so you compare like with like from run to run do

Code: Select all

sync
echo 3 > /proc/sys/vm/drop_caches 

at the start of each run. This will slow second and subsequent tests due to the 'cold' caches.

If you raid set is on rotating rust, the head/platter data limit will change by a factor of 4 or 5 from the outer to inner tracks. The drives are zoned. You get more sectors per track at the edge than near the spindle.
You should be able to hit that anywhere on the disk, unless its doing a resync.

Don't even think about using SMR drives in a raid set.

Has the raid6 paraty method changed between kernels?
I only use raid5 but

Code: Select all

 $ dmesg | grep -i raid6
[   10.254857] raid6: skipped pq benchmark and selected avx2x4
[   10.254857] raid6: using avx2x2 recovery algorithm

The kernel is supposed to choose the fastest pq calculator for your CPU

[Jimini]

Neddy & dmpogo, thank you for your replies.

I could run a few tests with bonnie++ while taking care for more constant basic conditions.
But at the moment I quite convinced, that changing kernel versions led to different performance results.

Code: Select all

[   13.732730] md/raid:md0: not clean -- starting background reconstruction

You can't do any useful speed tests with that going on.

You are absolutely right - but no process except tar was writing data to the array. And when I had to hard-reset the system, this was only because the system was stuck while writing veeeery slowly. Thus, the inconsisteny was very small, so the background reconstruction of the array was already finished, when I re-logged in.

Hopefully, you have a write intent bitmap so the reconstruction time is minimised?

Code: Select all

$ cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md127 : active raid5 sdc3[0] sda3[2] sdd3[3] sdb3[1]
      23441117184 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/59 pages [8KB], 65536KB chunk

Notice the 'bitmap' entry. It's not always useful, but every little helps. You have space for it in the mdadm header, so it costs nothing to add.

Yes, I have :)

Code: Select all

share2 ~ # cat /proc/mdstat 
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid6 sdf[3] sdb[0] sdd[6] sde[1] sdg[5] sda[2] sdc[4]
      78128737280 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/117 pages [0KB], 65536KB chunk

# mdadm -D /dev/md0 | grep bitmap

Code: Select all

Consistency Policy : bitmap

To flush your caches, so you compare like with like from run to run do

Code: Select all

sync
echo 3 > /proc/sys/vm/drop_caches 

at the start of each run. This will slow second and subsequent tests due to the 'cold' caches.

This is a good hint, I added this to my test script.

If you raid set is on rotating rust, the head/platter data limit will change by a factor of 4 or 5 from the outer to inner tracks. The drives are zoned. You get more sectors per track at the edge than near the spindle.
You should be able to hit that anywhere on the disk, unless its doing a resync.

Please keep in mind, that I this array is in use since 2023. Everything worked without problems until the middle of 2024 or so. It took some time until I noticed, that changing the kernel version was a workaround.

Don't even think about using SMR drives in a raid set.

Of course not! :)

Has the raid6 paraty method changed between kernels?
I only use raid5 but

Code: Select all

 $ dmesg | grep -i raid6
[   10.254857] raid6: skipped pq benchmark and selected avx2x4
[   10.254857] raid6: using avx2x2 recovery algorithm

The kernel is supposed to choose the fastest pq calculator for your CPU

For me it looks as if nothing had changed between 6.1.127, 6.6.74 and 6.12.14:

6.1.127:

Code: Select all

[    2.275772] raid6: avx2x4   gen() 36806 MB/s
[    2.445772] raid6: avx2x2   gen() 38635 MB/s
[    2.615772] raid6: avx2x1   gen() 30153 MB/s
[    2.615840] raid6: using algorithm avx2x2 gen() 38635 MB/s
[    2.785772] raid6: .... xor() 22290 MB/s, rmw enabled
[    2.785854] raid6: using avx2x2 recovery algorithm

6.6.74:

Code: Select all

share2 ~ # dmesg | grep -i raid6
[    2.283004] raid6: avx2x4   gen() 37004 MB/s
[    2.453005] raid6: avx2x2   gen() 38942 MB/s
[    2.623005] raid6: avx2x1   gen() 30282 MB/s
[    2.623073] raid6: using algorithm avx2x2 gen() 38942 MB/s
[    2.793005] raid6: .... xor() 22487 MB/s, rmw enabled
[    2.793086] raid6: using avx2x2 recovery algorithm

6.12.14:

Code: Select all

[    2.284097] raid6: avx2x4   gen() 37074 MB/s
[    2.454096] raid6: avx2x2   gen() 38892 MB/s
[    2.624097] raid6: avx2x1   gen() 30169 MB/s
[    2.624165] raid6: using algorithm avx2x2 gen() 38892 MB/s
[    2.794097] raid6: .... xor() 22496 MB/s, rmw enabled
[    2.794178] raid6: using avx2x2 recovery algorithm

As mentioned before, I would next try to test the performance with bonnie++, I should be able to post some results at the end of this week. If somebody has something to add, please feel free!

Best regards
Jimini

[pietinger]

[...] This script is now doing its job, like with kernel 6.1.

Seems like it is working again. [...]

<=>

[...] But if 6.12 is olso misbehaving, [...]

To be on the safe side, I would like to ask once again: Do I understand correctly that with 6.12. the test “Overwrite many small files” works correctly AGAIN and does NOT lead to a hang, as with 6.6. ?

If so, then I would say that there really was a kernel regression between 6.1 and 6.6 ... which was then fixed between 6.6 and 6.12. But finding that out by bisecting might be a herculean task.

[pietinger]

Yes, it is a monolithic, No-Multilib and headless Server:

Code: Select all

# CONFIG_IA32_EMULATION is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
# CONFIG_MODULES is not set
# CONFIG_DRM_AMDGPU is not set
# CONFIG_SOUND is not set

... but it is not a hardened one [*] ... e.g. ->

Code: Select all

CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SLAB_BUCKETS is not set
# CONFIG_SLUB_STATS is not set
CONFIG_SLUB_CPU_PARTIAL=y
# CONFIG_RANDOM_KMALLOC_CACHES is not set

(described in both articles: KSPP [*] and https://wiki.gentoo.org/wiki/User:Pieti ... nt_options )

... and you could disable some unused options, if you enable:

Code: Select all

# CONFIG_EXPERT is not set

->

Code: Select all

CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_HYGON=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_CPU_SUP_ZHAOXIN=y

See more here: https://wiki.gentoo.org/wiki/User:Pieti ... xpert_Mode

without this:

Code: Select all

# CONFIG_I2C_HID_ACPI is not set

... all these would never work (and you dont use them at all):

Code: Select all

CONFIG_HID_A4TECH=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
... and many more

(see also: https://wiki.gentoo.org/wiki/User:Pieti ... ID_Drivers )

*) I wrote this additional post because I would highly recommend this for a server:
https://wiki.gentoo.org/wiki/User:Pieti ... _with_KSPP

[dmpogo]

[...] This script is now doing its job, like with kernel 6.1.

Seems like it is working again. [...]

<=>

[...] But if 6.12 is olso misbehaving, [...]

To be on the safe side, I would like to ask once again: Do I understand correctly that with 6.12. the test “Overwrite many small files” works correctly AGAIN and does NOT lead to a hang, as with 6.6. ?

If so, then I would say that there really was a kernel regression between 6.1 and 6.6 ... which was then fixed between 6.6 and 6.12. But finding that out by bisecting might be a herculean task.

Well, if it is fixed in 6.12, then it is not as interesting, but perhaps just reading kernel ChangeLogs could shed a light. Unless fix was accidental

[NeddySeagoon]

Jimini,

Rule one is "Assume Nothing" but i had a feeling I might have been trying to teach my granny to suck eggs. :)

[Jimini]

Dear all,

thank you for your further replies and assistance.
@pietinger: I guess that tuning the kernel will be a bigger project for me - to avoid side effects, I will postpone this until I have ended my tests :)
@Neddy: haha, but yes, I agree that assuming nothing is the correct approach here.

To be on the safe side, I would like to ask once again: Do I understand correctly that with 6.12. the test “Overwrite many small files” works correctly AGAIN and does NOT lead to a hang, as with 6.6. ?

Yes, so I ran a few tests with bonnie++. The results for 6.1.127, 6.6.74 and 6.12.14 can be seen here: https://syncookie.de/bonnie/
To me, they look quite inconsistent and do not fit at all to the previous anomalies.

So I focused on the initial problem, namely overwriting an existing directory with very many small files.
The steps were always the same:

Code: Select all

sync
echo 3 > /proc/sys/vm/drop_caches
time tar -xzpf linux.tar.gz

Please note, that the target dir was not deleted, so tar always had to overwrite the dirs and files.

I noticed

6.1.127:
- 10 repetitions took about 1h46m. Unpacking took about 13s, but "sync" needed a lot more time.

6.6.74:
- I cancelled the extraction of the tarball after 2 hours, at that point repetition #2 out of 10 was going on. While having a look at the dirty memory (watch -n 1 grep -e Dirty: /proc/meminfo), I noticed that writing buffered data to disk happened at ~4K/s.

6.12.14:
- 10 repetitions took about 7 minutes. Unpacking was between 14s and 18s, the rest was syncing.

I am unsure how to proceed now. IMHO I should compare the kernel configs to ensure mostly identical kernels. Do you agree? Or do you have any ideas what else I could try? is there any additional information I could give?

Kind regards
Jimini