read and write files not at the same time [solved]

[SarahS93]

if i copy a big file, a 10 gig file from one hdd to an other hdd i see (in gkrellm) that the file is reading for 25...35 seconds, than stop reading, and than writing for 25...30 seconds. this goes on again and again bis the file i complete copied. read and write are not at the same time, is this normal? or have i any wrong configs? ?wrong kernel configs?

my system is a intel 8 core xeon cpu with 2x 32gb on a w480 board, and 6 sata hdd and on nvme ssd

here is my kerne .config file https://dpaste.com/7VHYVLEBL

edit
by copy big files to an sshfs mount i see read the file for some seconds for 10...20 seconds, and than there are stops for 10...20 seconds.
but read the file from hdd and send it to the sshfs mount (write) are done at the same time....

[ulenrich]

Since some major kernel versions I have the same issue with heavy io running slowly on a hard drive.

[tholin]

This has been a problem for years.

The simplified version of how things are meant to work is, when a process writes to disk that write is put in pagecache and the actual write happens later. When the cache is filled up to the "dirty_background_threshold" the kernel will begin doing writeout in the background. If the cache reaches the "dirty_threshold" the writing process is halted until the amount of dirty data in cache is under control. At least that is what the admin-guide/sysctl/vm.rst documentation suggests. By default those ratios are 10% and 20% of free memory.

https://github.com/torvalds/linux/blob/v5.14/mm/page-writeback.c#L817-L1069 here is some of the code involved. From the high comment/code ratio you can guess that the explanation above is an oversimplification. I've never bothered trying to wrap my head around exactly how it works but something goes wrong.

What typically happens is that written data is cached until the dirty_background_threshold is hit. Then the writing process is immediately halted and remains that way until almost all the dirty data is flushed. When it's unhalted it goes back to writing and the process repeats. We end up with oscillations between reading and writing. I've considered reporting this problem upstream but I know from experience that the VM folks are not that responsive to user reports so I've never bothered.

A workaround you can use it to set sysctl vm.dirty_background_bytes to a low value. I have mine set to 134217728. That will start the writeout sooner and for some reason dampen most of the oscillation. The downside is it will more aggressively write out temporary files that otherwise would never reach disk.

To change a sysctl value each boot make the change to /etc/sysctl.conf.

[SarahS93]

in
/etc/sysctl.conf
vm.dirty_background_bytes = 134217728
and than
sysctl -p
works fine for me, solved

thanks

!!!

[NeddySeagoon]

SarahS93,

That works between two different physical media.
Do you really want to do reads/writes at the same time, moving files between partitions on the same drive?
NCQ will help minimise the extra (wasted) head movements but read a RAM full, write a RAM full, is now the fastest strategy as head movements take most of the time.

Its a compromise. There is no single right answer.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.