AMD64 system slow/unresponsive during disk access (Part 2)

Message

timeBandit · Post by **timeBandit** » Sat Sep 19, 2009 12:29 am

Continuation of AMD64 system slow/unresponsive during disk access....

For convenience, in case this changeover happened in mid-conversation, the above link refers to the last page of the prior topic, not the first.

joi_ · Post by **joi_** » Sun Sep 20, 2009 9:47 pm

there's a new thread at LKML about responsiveness under I/O load
http://lkml.org/lkml/2009/9/19/288
speak up there people!

lagalopex · Post by **lagalopex** » Mon Sep 21, 2009 12:04 pm

I tried the BFS scheduler, but my usb system would afterwards not recognize any new devices.
So I have to wait a bit longer...

In the lkml thread they mentioned:

Code: Select all

for i in /sys/block/sd?/queue/iosched/quantum ; do echo 1 > "$i" ; done

quantum:
The amount of requests we select for dispatch when the driver asks for work to do and the current pending list is empty.
Default is 4.

Will now use it, but looks good so far.

kernelOfTruth · Post by **kernelOfTruth** » Mon Sep 21, 2009 1:43 pm

lagalopex wrote:I tried the BFS scheduler, but my usb system would afterwards not recognize any new devices.
So I have to wait a bit longer...

In the lkml thread they mentioned:
Code: Select all
for i in /sys/block/sd?/queue/iosched/quantum ; do echo 1 > "$i" ; done
quantum:
The amount of requests we select for dispatch when the driver asks for work to do and the current pending list is empty.
Default is 4.
Will now use it, but looks good so far.

I thought the higher the better (I'm currently using 64 or sometimes

but I will try 1

thanks !

kernelOfTruth · Post by **kernelOfTruth** » Thu Sep 24, 2009 10:02 pm

all of those recommendations of extremly lowering numbers of dirty_background_ratio, dirty_ratio, quantum, etc. seems to have made it magnitudes more worse !

some new stuff:

1) for this to work you need to switch to CFQ (if you use zen-sources or a kernel which has support for BFQ, better use that)

for i in /sys/block/sd*; do
/bin/echo "bfq" > $i/queue/scheduler
done

2) if you have buggy ncq-implementation in your drive-firmware (Seagate - I'm looking at you

)

set slice_idle to "0" - you don't need this with newer kernels and/or BFQ

so the code would look like:

for i in /sys/block/sd*; do
/bin/echo "bfq" > $i/queue/scheduler
/bin/echo "0" > $i/queue/iosched/slice_idle # default: 6, 0 fixes low throughput with drives which have a buggy ncq implementation
done

also enable:

# "the following practically disables NCQ which is buggy"
# "on a lot of drives and known to drive CFQ and other i/o schedulers crazy

"

for i in /sys/block/sd*; do
/bin/echo "2" > $i/device/queue_depth
done

(you can also set it to "1" - 0 doesn't work)

3)

set up portage

Code: Select all

PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

in /etc/make.conf

setting up your daemons

execute:

for i in `pidof kjournald` ; do ionice -c3 -p $i ; done
for i in `pidof kjournald2` ; do ionice -c3 -p $i ; done
for i in `pidof pdflush` ; do renice 10 $i ; done
for i in `pidof kcryptd` ; do ionice -c1 -p $i ; done

if the above ionice-settings breaks anything (if at all) please report !

4) renice X or some of X

Code: Select all

for i in `pidof X` ; do renice -10 $i ; done
for i in `pidof kwin` ; do renice -10 $i ; done

...

this doesn't fix all hickup for me but improves interactivity A LOT

thanks for reading until now !

enjoy !

mamunata · Post by **mamunata** » Fri Sep 25, 2009 11:50 am

I follow this topic for a while, because I have a laptop with Turion64 CPU and 64bit gentoo on it, and I/O performance is terrible - system load increases to 10 or even 15-20 on heavy disk usages and system is completely unusable.
I've been impressed that there's different "tricks" that decrease the problem and lot of them are for SATA (sd*) hard disks (with or without NCQ), but my laptop is with IDE hard disk and problem is fully present.
So am I missing something or problem is independent of hard disk type and is inherent for all types HDD?

Elv13 · Post by **Elv13** » Sat Sep 26, 2009 8:20 pm

Try the kernel 2.6.30-rc5, it is a version just after a patch that really worked well and a redesign that broke it again. I use sata drives, but it may work for you too.

sidamos · Post by **sidamos** » Sat Sep 26, 2009 10:09 pm

I am having the same problem since 2.6.30 and I also do not have SATA. 2.6.29 works better. But this is not fixed in 2.6.31, as far as I understood.

kernelOfTruth · Post by **kernelOfTruth** » Sun Sep 27, 2009 2:53 pm

mamunata wrote:I follow this topic for a while, because I have a laptop with Turion64 CPU and 64bit gentoo on it, and I/O performance is terrible - system load increases to 10 or even 15-20 on heavy disk usages and system is completely unusable.
I've been impressed that there's different "tricks" that decrease the problem and lot of them are for SATA (sd*) hard disks (with or without NCQ), but my laptop is with IDE hard disk and problem is fully present.
So am I missing something or problem is independent of hard disk type and is inherent for all types HDD?

those are more than 1 problem,

*) the problem with low throughput is related to a buggy firmware on several SATA-drives with NCQ

*) the problem with lagging / bad desktop interactivity is (seems) specific to the x86_64 / amd64 architecture and cfq and cfs (the cpu-scheduler) seem to be involved

there are more:

*) extremely low throughput with enabled barriers

*) ...

sidamos · Post by **sidamos** » Sun Sep 27, 2009 3:11 pm

sidamos wrote:I am having the same problem since 2.6.30 and I also do not have SATA. 2.6.29 works better. But this is not fixed in 2.6.31, as far as I understood.

Additional info: I am running AMD64X2 CPU with 32 bit Gentoo.

krinn · Post by **krinn** » Sun Sep 27, 2009 3:16 pm

sidamos wrote:
sidamos wrote:I am having the same problem since 2.6.30 and I also do not have SATA. 2.6.29 works better. But this is not fixed in 2.6.31, as far as I understood.
Additional info: I am running AMD64X2 CPU with 32 bit Gentoo.

so the problem is beyond AMD64 (64bits) version ? maybe some title change so... As is it seems to refer to the arch AMD64 and not the cpu ?

Cyker · Post by **Cyker** » Sun Sep 27, 2009 7:08 pm

krinn wrote:
sidamos wrote:
sidamos wrote:I am having the same problem since 2.6.30 and I also do not have SATA. 2.6.29 works better. But this is not fixed in 2.6.31, as far as I understood.
Additional info: I am running AMD64X2 CPU with 32 bit Gentoo.
so the problem is beyond AMD64 (64bits) version ? maybe some title change so... As is it seems to refer to the arch AMD64 and not the cpu ?

Well, on going from 2.6.28 to 2.6.30, I've been experiencing the total opposite!

I'm running standard 32-bit x86 on an Opteron 180 and am getting much lower iowait and much better throughput on my HDs and network connections.

nero37 · Post by **nero37** » Mon Sep 28, 2009 11:40 am

kernelOfTruth wrote:4) renice X or some of X
Code: Select all
for i in `pidof X` ; do renice -10 $i ; done
for i in `pidof kwin` ; do renice -10 $i ; done

Where is the best place to have this run automatically at start-up? The window manager starts too late to renice it from a init.d script and since it requires root privileges can't easily be done after login.

lagalopex · Post by **lagalopex** » Tue Sep 29, 2009 2:01 pm

Anybody else running bfs?
I am currently running a 2.6.31.1 with the bfs-240 patch applied.
It now works for me and is pretty responsive as far as I can tell

kernelOfTruth · Post by **kernelOfTruth** » Wed Sep 30, 2009 4:50 pm

lagalopex wrote:Anybody else running bfs?
I am currently running a 2.6.31.1 with the bfs-240 patch applied.
It now works for me and is pretty responsive as far as I can tell

++

anyone who is saying that this isn't the i/o schedulers / VFS' / the cpu schedulers (CFS') fault has NO idea

the most responsive kernel for me (without BFS) so far was a 2.6.24.something based zen-kernel

could have been that it had RSDL applied / ported - unfortunately I don't have the patch or sources for it anymore ...

anyways:

try 2.6.31-zen2 with BFS (build 240) enabled, there's almost no lagging anymore - additionally the desktop is much much more responsive

lagalopex · Post by **lagalopex** » Thu Oct 01, 2009 9:43 am

With bfs240 I had again a keyboard issue. So I switched to 2.6.32-rc1 as stated in [topic=794535]Kernel 2.6.30/31 desktop interactivity patch (no BFS)[/topic] some improvements for cfs got merged in mainline. Its imo better...

bfs300 is now available... will try it the next days...

Update: Now with bfs302 as the 300 still had a keyboard issue for me. ck noted that switching to evdev helps... I will see...

Update2: bfs303 and ck ran a new test against vanilla-2.6.32-rc3. The rc is doing well

MageSlayer · Post by **MageSlayer** » Sun Oct 11, 2009 7:04 am

I confirm that problem with keyboard under Xorg + bfs has gone after switching to evdev.
evdev 2.1.3 version is the only one which does not have buggy keys repetition though.

Lepaca Kliffoth · Post by **Lepaca Kliffoth** » Mon Oct 12, 2009 7:29 pm

BFS solves the problem completely for me. It doesn't make it better; under the kind of loads that used to still create problems even with the few kernel versions that were better, the mouse jerkiness and frame skipping in videos simply doesn't appear.

Things that used to be problematic and now are fine:

- copy stuff over SMB
- check out linux git sources
- paludis -s

It is now impossible for me to reproduce the problem.

I'm using the latest zen-sources.

Sujao · Post by **Sujao** » Wed Oct 21, 2009 5:23 pm

Lepaca Kliffoth wrote:BFS solves the problem completely for me. It doesn't make it better; under the kind of loads that used to still create problems even with the few kernel versions that were better, the mouse jerkiness and frame skipping in videos simply doesn't appear.

Things that used to be problematic and now are fine:

- copy stuff over SMB
- check out linux git sources
- paludis -s

It is now impossible for me to reproduce the problem.

I'm using the latest zen-sources.

Does it also solve the dd if=/dev/zero of=dump problem where the system locks up as soon as RAM is full?

Where can I get zen-sources? zen-sources.org seems to be down.

forkboy · Post by **forkboy** » Wed Oct 21, 2009 10:42 pm

Where can I get zen-sources? zen-sources.org seems to be down.

See www.zen-kernel.org and this thread.

wazoo42 · Post by **wazoo42** » Wed Oct 28, 2009 1:20 am

They are available in portage (my portage tree shows .31-r4 is the latest one), as well as from the overlay, zen-sources. I found BFS helps, but I'm not sure it eliminated the problem. Then again, I haven't done any sort of concrete testing...

Sujao · Post by **Sujao** » Wed Oct 28, 2009 4:59 am

... no can't be true ....

.....

...

....

...

Oh my God. It actually seems to work. I tried out the things that were annoying me for almost 2 years now and my system remained responsive. Usually when I write a big (>5GB) file with mmg to harddrive or copy them click response time in most applications is between 10-30s. mmg window doesn't get refreshed until its finished after several minutes and videos don't play proberly. Now I can surf easily with firefox, even play a movie with no visible delay and the mmg window is updating as expected. I'll report back if something changes....I still can't really believe that this is true.

EDIT: It's a shame the vanilla kernel doesn't accomplish this. Thanks to the ZEND Guys! I love you right now!

EDIT2: Moving a 8GB file still increases the response time of applications. For example smplayer open file dialog reacts after 3-4s but this is acceptable. Still much better then 30s. And video still plays smoothly.

mamunata · Post by **mamunata** » Wed Oct 28, 2009 5:18 pm

Sujao wrote: ... no can't be true .... ..... ... .... ... Oh my God. It actually seems to work. I tried out the things that were annoying me for almost 2 years now and my system remained responsive. Usually when I write a big (>5GB) file with mmg to harddrive or copy them click response time in most applications is between 10-30s. mmg window doesn't get refreshed until its finished after several minutes and videos don't play proberly. Now I can surf easily with firefox, even play a movie with no visible delay and the mmg window is updating as expected. I'll report back if something changes....I still can't really believe that this is true.

EDIT: It's a shame the vanilla kernel doesn't accomplish this. Thanks to the ZEND Guys! I love you right now!

EDIT2: Moving a 8GB file still increases the response time of applications. For example smplayer open file dialog reacts after 3-4s but this is acceptable. Still much better then 30s. And video still plays smoothly.

Can you post what options did you compile kernel with? I'm running zen-kernel-2.6.31-r4 and performance is not much better.
I've noticed (but not 100% sure) that if I leave my PC alone, and not working on it for a couple of hours, when I get back the PC is very slow.

kernelOfTruth · Post by **kernelOfTruth** » Wed Oct 28, 2009 6:10 pm

mamunata wrote:
Sujao wrote: ... no can't be true .... ..... ... .... ... Oh my God. It actually seems to work. I tried out the things that were annoying me for almost 2 years now and my system remained responsive. Usually when I write a big (>5GB) file with mmg to harddrive or copy them click response time in most applications is between 10-30s. mmg window doesn't get refreshed until its finished after several minutes and videos don't play proberly. Now I can surf easily with firefox, even play a movie with no visible delay and the mmg window is updating as expected. I'll report back if something changes....I still can't really believe that this is true.

EDIT: It's a shame the vanilla kernel doesn't accomplish this. Thanks to the ZEND Guys! I love you right now!

EDIT2: Moving a 8GB file still increases the response time of applications. For example smplayer open file dialog reacts after 3-4s but this is acceptable. Still much better then 30s. And video still plays smoothly.
Can you post what options did you compile kernel with? I'm running zen-kernel-2.6.31-r4 and performance is not much better.
I've noticed (but not 100% sure) that if I leave my PC alone, and not working on it for a couple of hours, when I get back the PC is very slow.

you need to select BFS (as a CPU scheduler instead of the stock-scheduler CFS) and select BFQ for the i/o-scheduler

kernel-config:

Code: Select all

# CONFIG_CPU_CFS is not set
CONFIG_CPU_BFS=y
CONFIG_CPU_BFS_AUTOISO=y
# CONFIG_BFS_CUSTOM_RR is not set

Code: Select all

for i in /sys/block/sd*; do
         /bin/echo "bfq" >  $i/queue/scheduler
done

the following might also provide you with additional throughput:

Code: Select all

for i in /sys/block/sd*; do
         /bin/echo "256" >  $i/queue/read_ahead_kb
         /bin/echo "192" >  $i/queue/max_sectors_kb
         /bin/echo "1"   >  $i/queue/rq_affinity
         /bin/echo "0"   >  $i/queue/nomerges
done

Sujao · Post by **Sujao** » Wed Oct 28, 2009 6:13 pm

Well it's not as good as in the first test. This time I had to hash a ~6GB file and there were big delays. I had a flashvideo in firefox running and another movie in PAL and the hash. Firefox didn't react to clicks for more than 1min until the hash was finished, the flash video ran with no problems, the movie stopped about every 15s for 1-2s but i think with the old kernel it was much worse, the movie would freeze completely and run for 1-2s every 15s.

Overall it seems to be better but I didn't make any identical benchmarks. I simply try things that used to lock my system. So maybe my first test was somehow different and that is why it ran so flawlessly.

My kernel configuration (The forum doesn't seem to accept such a long text):
http://pastebin.com/f78f5fec1

EDIT:

CONFIG_CPU_BFS_AUTOISO=y was not set for me, recompiling at the moment. What does it do exactly? Almost no help available. Scheduler for the X-Server?

Code: Select all

for i in /sys/block/sd*; do
         /bin/echo "bfq" >  $i/queue/scheduler
done

Was not necessary for me. All drives have the same output:

Code: Select all

# cat /sys/block/sda/queue/scheduler
noop fifo anticipatory deadline cfq vr [bfq]

mamunata wrote:I've noticed (but not 100% sure) that if I leave my PC alone, and not working on it for a couple of hours, when I get back the PC is very slow.
I think this is not caused by a faulty IO scheduler. I have the same behaviour. I think the RAM is simply cleaned of not accessed desktop applications and when you come back it needs to reread libraries from the harddrive. Can anyone confirm that?

EDIT2: OK, seriously..

.... Sorry guys. This is the second time I thought the problem was kind of solved. Now, I did three "comparable" benchmarks with my old 2.6.30-gentoo-r1(1), linux-2.6.31-zen4 CONFIG_CPU_BFS_AUTOISO=n(2) and linux-2.6.31-zen4 CONFIG_CPU_BFS_AUTOISO=y(3). The benchmark: Dumping a file from hdd A to hdd B, vieweing a movie from hdd A and hdd C, viewing the same flashvideo and hashing a file from hdd A and hdd B with sha256sum.

The result (just my subjective perception): 2 and 3 makes no difference, 1 is a little worse, but not as worse as I would have expected. I think I need a more detailed automatic benchmark with measurable results to make further statements, as my previous claims turned out to be false.