AMD64 system slow/unresponsive during disk access...

[wazoo42]

Jens Axboe's patch (post 397 in the link MageSlayer gave above) worked wonders for my amd64 install (dual core opteron) with vanilla 2.6.30.4. I'll work on getting some concrete numbers, hopefully they'll backup the improved responsiveness I see in KDE4.

[devsk]

Jens Axboe's patch (post 397 in the link MageSlayer gave above) worked wonders for my amd64 install (dual core opteron) with vanilla 2.6.30.4. I'll work on getting some concrete numbers, hopefully they'll backup the improved responsiveness I see in KDE4.

That's interesting! Are you using the patch with NCQ or without. Can you post the output of:

Code: Select all

cat /sys/block/*/queue/nr_requests

[MageSlayer]

Yes. It would be great if you could provide some interbench numbers.

[wazoo42]

It appears I'm using NCQ and the CFQ i/o scheduler.

Code: Select all

benjfitz linux # cat /sys/block/*/queue/nr_requests
128
128
128
128
128
128
128
128

benjfitz linux # cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

[devsk]

It appears I'm using NCQ and the CFQ i/o scheduler.

Code: Select all

benjfitz linux # cat /sys/block/*/queue/nr_requests
128
128
128
128
128
128
128
128

benjfitz linux # cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

That's in contrast with what other people are seeing. hmmm...

[3Towers]

I have some more for you to test. For quite some time I had the same problems (unresponsive UI during IO operations and an almost completely freezed system when I was trying to rip a DVD with read errors). I tried several options already discussed in this thread (including the kernel ncq patch) with no success. Today I built a completely new kernel with the following options changed:
- disabled SMT (HyperThreading support) as my Q6700 doen't has HT
- enabled preemptile RCU
- enabled low latency desktop (forced preemption)
- disabled the IDE layer (only compiled libata SATA/PATA drivers)
additionally I swtched SATA mode in my BIOS to AHCI (IDE mode before) and set the RTC to 64bit.

With these settings all my problems are gone now! Only drawback: I'm not able to boot up Windowx XP with these BIOS settings if I need it again. *g*
When I have some more time I will try to figure out which option is responsible for this success and post it here.

Hope this can help some of you!

[energyman76b]

disable smt and ahci probably

I don't have forced preemption nor rcu preemption and I don't suffer from long lags anymore. For a long time.

[kernelOfTruth]

disable smt and ahci probably

I don't have forced preemption nor rcu preemption and I don't suffer from long lags anymore. For a long time.

I got to test that with smt, thanks !

you don't need ahci anyways with your SCSI disks

[energyman76b]

disable smt and ahci probably

I don't have forced preemption nor rcu preemption and I don't suffer from long lags anymore. For a long time.

I got to test that with smt, thanks !

you don't need ahci anyways with your SCSI disks

my superfast scsi disks are in my other box

[luispa]

For those having this problem, especially if they have SATA drives, it would probably be worth a shot to try the deadline scheduler instead of cfq.

Everything I've read over the last year or so seems to indicate there is still an I/O problem with cfq on some systems, and also that generally with SATA drives deadline is often a better scheduler that cfq. Kernel >=2.6.30-rc4 seemed to improve it somewhat (as mentioned), but I'm still sticking with deadline myself until I'm convinced this is really fixed with cfq.

You need to enable support in your kernel (probably already has it, but check your .config file). If not, you'll need to recompile your kernel and enable deadline, but if it does already have it, just append your grub kernel line with

Code: Select all

 elevator=deadline

and reboot.
If it makes a difference great, but if not, just remove the append.

wrc1944, thanks for the update. I spend some time with .30-rc4 but still suffering the issue I mentioned...

@fangorn
Thanks for the information, as I said here is the result with 2.6.29: no problem, back to normal behaviour. I'm not suffering problems with I/O now. Obviously I cant add any value here, but my experience. 2.6.28: Ok, 2.6.30: I/O issue, 2.6.29: Ok.
Luis

I'm going to try with your suggestion (deadline) and report back.

Luis

[luispa]

I'm going to try with your suggestion (deadline) and report back.

Luis

My report:

I've noticed now with .30 & deadline the system is more responsive but still seeing that disk writtes (syncing) are somehow delayed meaning sporadic reponsiveness, so in general I feel that the system is more reliable with .29.

With interbench didn't notice the difference. I've run it under runlevel 3 though, maybe I should run it under runlevel 1 as recommended.

Here is the output:

Code: Select all

bolica interbench-0.30 # cat 2.6.29-gentoo-r5.log 

Using 805133 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.29-gentoo-r5 at datestamp 200908270935

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.013 +/- 0.088       1.93		 100	        100
Video	  0.006 +/- 0.00695    0.015		 100	        100
X	   0.01 +/- 0.061       1.47		 100	        100
Burn	  0.022 +/- 0.261       5.79		 100	        100
Write	   0.01 +/- 0.0375     0.733		 100	        100
Read	  0.012 +/- 0.0677      1.64		 100	        100
Compile	  0.111 +/- 0.983       16.6		 100	        100
Memload	  0.021 +/- 0.088       1.52		 100	        100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.007 +/- 0.0205     0.809		 100	        100
X	  0.015 +/- 0.138       3.45		 100	        100
Burn	  0.016 +/- 0.152       4.17		 100	        100
Write	  0.007 +/- 0.0153     0.481		 100	        100
Read	  0.008 +/- 0.0144     0.406		 100	        100
Compile	  0.023 +/- 0.357       10.9		 100	        100
Memload	  0.017 +/- 0.027      0.668		 100	        100

--- Benchmarking simulated cpu of X in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	      0 +/- 0.00111    0.013		 100	        100
Video	      0 +/- 0.000812    0.01		 100	        100
Burn	  0.533 +/- 2.57          18		86.2	       84.4
Write	      0 +/- 0.00153    0.017		 100	        100
Read	      0 +/- 0.0014     0.018		 100	        100
Compile	  0.446 +/- 2.16          14		92.6	       89.7
Memload	      0 +/- 0.00165    0.018		 100	        100

--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU
None	      0 +/- 0              0		 100
Video	      0 +/- 0              0		 100
X	      0 +/- 0              0		 100
Burn	   1.35 +/- 4.43        19.7		98.7
Write	      0 +/- 0              0		 100
Read	      0 +/- 0              0		 100
Compile	   4.89 +/- 8.49        34.1		95.3
Memload	      0 +/- 0              0		 100

Code: Select all

bolica interbench-0.30 # cat 2.6.30-gentoo-r4.log 

Using 805133 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.30-gentoo-r4 at datestamp 200908271023

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.007 +/- 0.00727    0.015		 100	        100
Video	  0.005 +/- 0.00515    0.011		 100	        100
X	  0.006 +/- 0.00629    0.012		 100	        100
Burn	  0.015 +/- 0.107       1.93		 100	        100
Write	  0.008 +/- 0.00859    0.017		 100	        100
Read	  0.008 +/- 0.00939    0.098		 100	        100
Compile	  0.005 +/- 0.0054     0.014		 100	        100
Memload	  0.021 +/- 0.122       2.45		 100	        100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.005 +/- 0.00553    0.015		 100	        100
X	  0.005 +/- 0.00572    0.019		 100	        100
Burn	  0.021 +/- 0.227       5.66		 100	        100
Write	  0.006 +/- 0.00943     0.24		 100	        100
Read	  0.006 +/- 0.00627    0.017		 100	        100
Compile	  0.277 +/- 2.62        35.3		99.8	         99
Memload	  0.016 +/- 0.0319     0.719		 100	        100

--- Benchmarking simulated cpu of X in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	      0 +/- 0.000983   0.012		 100	        100
Video	      0 +/- 0.000592   0.008		 100	        100
Burn	  0.903 +/- 3.25          14		  89	       84.6
Write	      0 +/- 0.00117    0.012		 100	        100
Read	      0 +/- 0.00119    0.017		 100	        100
Compile	   1.04 +/- 4             28		85.6	         81
Memload	      0 +/- 0.00156    0.021		 100	        100

--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU
None	      0 +/- 0              0		 100
Video	      0 +/- 0              0		 100
X	      0 +/- 0              0		 100
Burn	   1.38 +/- 4.24        14.3		98.6
Write	      0 +/- 0              0		 100
Read	      0 +/- 0              0		 100
Compile	   3.35 +/- 7.49        34.4		96.8
Memload	      0 +/- 0              0		 100

Thanks,
Luis

[sidamos]

I have the same problem with AMD64x2 5400+ (running Gentoo 32 bit) and PATA disks. Also, they started with 2.6.30, AFAIK. I have low latency desktop on. I have tried cfq and anticipatory scheduler. However, IMHO this should not be relevant if the heavy IO is on another disk (in my case), see http://forums.gentoo.org/viewtopic-p-5958024.html

[devsk]

something brewing in the 2.6.31 release, which may be good news. anyone running 2.6.31 RCs here?

http://www.techworld.com.au/article/317 ... ux_desktop

[sidamos]

something brewing in the 2.6.31 release, which may be good news. anyone running 2.6.31 RCs here?

http://www.techworld.com.au/article/317 ... ux_desktop

This link does not work.

[devsk]

something brewing in the 2.6.31 release, which may be good news. anyone running 2.6.31 RCs here?

http://www.techworld.com.au/article/317 ... ux_desktop

This link does not work.

I just clicked on in your post and it did work! anyway, here is the relevant part:

The kernel developers have been working on improvements to desktop interactivity, particularly when it's under memory pressure since the last release, version 2.6.30, in June.

Desktop applications can experience long and noticeable pauses when the application's code path jumps to a part of the code that is not cached in memory and needs to be read from the disk, which is slower.

However, recent kernel memory management scalability work can result in a desktop environment with poor interactivity as applications become unresponsive too easily.

In version 2.6.31, some heuristics have been used to make it much harder to move the “mapped executable pages” out of the list of active pages, according to Kernelnewbies.org.

“The result is an improved desktop experience; benchmarks on memory tight desktops show clock time and major faults reduced by 50 per cent, and pswpin numbers (memory reads from disk) are reduced to about one-third. That means X desktop responsiveness is doubled under high memory pressure.”

Furthermore, memory flushing benchmarks in a file server shows the number of major faults going from 50 to 3 during 10 per cent cache hot reads.

[keenblade]

something brewing in the 2.6.31 release, which may be good news. anyone running 2.6.31 RCs here?

I have 2.6.31-rc5 here. Here the interbench resulsts:

Code: Select all

Using 979314 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.31-rc5 at datestamp 200909060950

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.085 +/- 0.379       6.64		 100	        100
Video	  0.122 +/- 0.652       11.8		 100	        100
X	  0.208 +/- 1.29        16.3		 100	        100
Burn	  0.035 +/- 0.211       4.15		 100	        100
Write	  0.255 +/- 1.36        21.5		 100	        100
Read	  0.117 +/- 0.875       13.6		 100	        100
Compile	   0.04 +/- 0.383       8.73		 100	        100
Memload	  0.024 +/- 0.101       2.29		 100	        100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.101 +/- 1.2         31.9		99.9	       99.7
X	  0.275 +/- 2.32        35.8		99.4	       99.2
Burn	  0.072 +/- 1.97        81.8		99.9	       99.9
Write	  0.242 +/- 2.6         67.3		99.7	       99.4
Read	  0.093 +/- 0.926       32.4		 100	       99.9
Compile	  0.076 +/- 0.881       27.1		 100	       99.9
Memload	  0.074 +/- 0.486       16.7		 100	       99.9

--- Benchmarking simulated cpu of X in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	      0 +/- 0.00419    0.063		 100	        100
Video	      0 +/- 0.00283    0.037		 100	        100
Burn	   6.15 +/- 36.5         434		52.5	       51.1
Write	      0 +/- 0.00193    0.023		 100	        100
Read	      0 +/- 0.00198    0.025		 100	        100
Compile	   9.32 +/- 39.4         408		33.5	       29.8
Memload	      0 +/- 0.00567    0.068		 100	        100

--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU
None	  0.016 +/- 0.395       9.57		 100
Video	  0.428 +/- 5.17         103		99.6
X	  0.355 +/- 5.11         103		99.6
Burn	   59.1 +/- 140          952		62.9
Write	  0.132 +/- 1.33        22.9		99.9
Read	  0.097 +/- 1.77        38.2		99.9
Compile	   49.3 +/- 89.1         911		  67
Memload	      0 +/- 0              0		 100

Note : I don't know how to interpret these results, but it seemed slow to me. Kernel is from linus git. No patch used and I have ccache, if it matters.

[devsk]

I think its the relative interpretation which may be useful. If you have 2.6.30 (or any other older) kernel, then boot into it and re-run interbench and let's see what numbers it throws up.

[Elv13]

In slashdot today:
http://linux.slashdot.org/story/09/09/0 ... art_pos=20

[d-fens]

sounds promising, a pity that i can't try it this week - but one question remains for me:

does the optimized (reduced) disk access just work around the root problem, the slow disk access starting from 2.18?

[BitJam]

I've been running amd64 for a couple of years and here is what I've noticed. As long as I had only one hard drive on the system, I never had any problem. Then as soon as I got a 2nd hard drive, transfer of large files from one drive to the other would be dog slow.

I've been recently playing with LiveUSBs and I've noticed something very strange. Transfer of large files to the usb drive is fairly fast if the usb is formated with fat32 but when I format the usb with ext2, the transfers become dog slow, such as 0.5 megabytes per second.

I've seen this on two out of two usb drives I've tested.

[devsk]

2.6.31 goes stable. Will try it later tomorrow. Anybody else up for some benchmarking and comparisons with respect to this bug?

[mamunata]

2.6.31 goes stable. Will try it later tomorrow. Anybody else up for some benchmarking and comparisons with respect to this bug?

I've compiled 2.6.31 but almost no difference:
-hdparm shows even slower speed - about 20MB/s on my laptop (wit 2.6.30 was 25MB/s)
-on heavy disk load desktop environment responds slowly and system load is 2-3
-swap is used rarely

[devsk]

2.6.31 goes stable. Will try it later tomorrow. Anybody else up for some benchmarking and comparisons with respect to this bug?

I've compiled 2.6.31 but almost no difference:
-hdparm shows even slower speed - about 20MB/s on my laptop (wit 2.6.30 was 25MB/s)
-on heavy disk load desktop environment responds slowly and system load is 2-3
-swap is used rarely

Thanks for replying because I was gonna do it today. I think I am gonna just pass the 2.6.31 release then.

[Elv13]

So far so good for me! Kernel 2.6.30 was a huge improvement over 2.6.2[0-9] and 2.6.31 seem to ba as stable and fast. And I finally removed debugging support from my kernel, I hope to never have it enable again :p. I think this solve the issue in my case, 64bit Linux is now exploiting the whole potential of my computer!

EDIT: Ignore my comment, skip this kernel, it suck, I am reverting to 2.6.30 right now.

[devsk]

So far so good for me! Kernel 2.6.30 was a huge improvement over 2.6.2[0-9] and 2.6.31 seem to ba as stable and fast. And I finally removed debugging support from my kernel, I hope to never have it enable again :p. I think this solve the issue in my case, 64bit Linux is now exploiting the whole potential of my computer!

EDIT: Ignore my comment, skip this kernel, it suck, I am reverting to 2.6.30 right now.

what happened? elaborate please.