AMD64 system slow/unresponsive during disk access (Part 2)

[CarstenIQ]

I ended up in using the pf-sources (3.5.2) with the BFQ patch set and zram. My 4 core Phenom II works smooth in x86_64 environment.
I works so good that the Flash player doesn't even crash anymore in Opera. For me the gentoo-sources (3.4.9 ) was just to sluggish and had too many lock-ups. It might be a considered a stable kernel but for me it is not usable kernel in x86_64 environment.

[yoshi314]

btw, running vanilla kernel i noticed that unrar command is incredibly slow. (similariy when unpacking the file with 7z)

unpacking a 3GB archive took well over an hour. is anyone else catching the same issue when not doing any performance improving workarounds?

edit: it stopped happening recently, for some reason.

[smlbstcbr]

Bump. Using kernel 3.5.7 and the deadline scheduler the problem is greatly reduced. Tough also modifying the PCI latency timer seems to have had a significant impact in the responsiveness of the whole system.

[_______0]

... PCI latency timer seems to have had a significant impact in the responsiveness of the whole system.

how to do this?? What's better higher values on all pci slots or something?

Also why on earth does setpci use hex values instead of more user-friendly NUMBERS???!!

where xx:yy.z are the first numbers listed by lspci, and n is a hex value between 0 and ff (setpci will round down ff, which is actually 255, to the required 248).

dis crazy... Ideally it'd be nice to have a web interface for setpci controlled by pretty sliders for each pci slot and with css shadows.

And for maximum priority for sound card (to prevent xruns) which value would be? Something like this?

Code: Select all

setpci -v -s 06:05.0 latency_timer=ff

thanks.

I forgot to add, how do intenger values between 0 and 248 translate into hex values? It be nice to have it as reference. This is confusing because according to a blog setpci hex values are ODD numbers instead of even oO

If anybody know this I need a table of the exact ODD setpci hex value with the corresponding BIOS even values, like this.

hex | hex odd intenger | bios even intenger
ff 255 248

here is the referenced site http://www.mythtv.org/wiki/PCI_Latency


And to expand on this topic. How can person determine which is the best combination of latency for all devices in order for a computer to run optimal?? Let's say I know now that I want to prevent the sound card to have xruns the set it to maximum priority, then for recording the hard drive would need an optimal value too. It's not accurate just to set something high because it 'feels' it'll run better.

[smlbstcbr]

The PCI timer is set in the BIOS setup options (each BIOS has a different setup, though should be very similar).
For INT to HEX: http://www.statman.info/conversions/hexadecimal.html

[yoshi314]

most of you probably know this, but it won't hurt to have links in the thread, for reference.

anyone brave enough to tackle latest 3.9 git?

this is said to help with original issue : http://git.kernel.org/cgit/linux/kernel ... 0911d95cf6

and more info is here : https://bugzilla.kernel.org/show_bug.cgi?id=12309#c621

[mack1]

I have installed git-sources-3.9-r3 and high latency is greatly reduced under heavy i/o operations !

[smlbstcbr]

Really?! I had noticed a significant improvement by using the RCU option in 3.7.10 with the deadline scheduler.

[depontius]

Really?! I had noticed a significant improvement by using the RCU option in 3.7.10 with the deadline scheduler.

I've at least heard of the deadline scheduler, though I use the default CFQ. I've also read of RCU, but I thought that was just plain baked-in as a feature to be used by other subsystems, especially handy for multiprocessor.

What is "the RCU option"?

[smlbstcbr]

Really?! I had noticed a significant improvement by using the RCU option in 3.7.10 with the deadline scheduler.

I've at least heard of the deadline scheduler, though I use the default CFQ. I've also read of RCU, but I thought that was just plain baked-in as a feature to be used by other subsystems, especially handy for multiprocessor.

What is "the RCU option"?

https://www.kernel.org/doc/Documentation/RCU/rcu.txt

[kernelOfTruth]

Really?! I had noticed a significant improvement by using the RCU option in 3.7.10 with the deadline scheduler.

I've at least heard of the deadline scheduler, though I use the default CFQ. I've also read of RCU, but I thought that was just plain baked-in as a feature to be used by other subsystems, especially handy for multiprocessor.

What is "the RCU option"?

https://www.kernel.org/doc/Documentation/RCU/rcu.txt

please post the CONFIG_...

option from your /usr/src/linux/.config file you're referring to

thanks

[smlbstcbr]

Here it is

Code: Select all

# RCU Subsystem
#
CONFIG_TINY_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500

[Mr.EVIL]

it was fixable long time ago with "realtime" patches...

[kernelOfTruth]

Here it is

Code: Select all

# RCU Subsystem
#
CONFIG_TINY_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500

thanks !


I believe though that most of us here already us a preemptible implementation of RCU

it was fixable long time ago with "realtime" patches...

unfortunately not an option with ZFS (?)

do you use virtualbox ? does that work for you with the rt-kernel ?

[depontius]

Here it is

Code: Select all

# RCU Subsystem
#
CONFIG_TINY_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500

Just a side-note... The CONFIG_PREEMPT_RCU option doesn't even show unless you're building a preemptible kernel. I normally build with "voluntary preemption," but this weekend when building 3.8.4 I went the pre-emptible route, and ticked CONFIG_PREEMPT_RCU=y. I don't think I ticked CONFIG_TINY_PREEMPT_RCU - I really need to go back and read if I should have used it for a mere dual-core CPU.

EDIT - a few typos, a little clarification.

[kernelOfTruth]

The PCI timer is set in the BIOS setup options (each BIOS has a different setup, though should be very similar).
For INT to HEX: http://www.statman.info/conversions/hexadecimal.html

now giving the pci latency timer approach a try:

http://web.archive.org/web/200808291627 ... l-hw2.html


will see if there's a difference with sound or not - thanks !

[ryao]

it was fixable long time ago with "realtime" patches...

unfortunately not an option with ZFS (?)

do you use virtualbox ? does that work for you with the rt-kernel ?

Why is that? If there is an incompatibility, I would like a bug report.

[kernelOfTruth]

it was fixable long time ago with "realtime" patches...

unfortunately not an option with ZFS (?)

do you use virtualbox ? does that work for you with the rt-kernel ?

Why is that? If there is an incompatibility, I would like a bug report.

from what I saw there are already at least 2 issues on github referring to kernels

one of them being: https://github.com/zfsonlinux/spl/issues/218

edit:

https://github.com/zfsonlinux/spl/issues/47

edit.


I get similar/same errors either building it patched into the kernel or as ebuilds as external modules (with 3.8.4 / 3.8-patched rt-kernel)


thanks for your continued support


edit:

that probably would be the only issue right now preventing me to use rt-kernels

edit.

[ryao]

from what I saw there are already at least 2 issues on github referring to kernels

one of them being: https://github.com/zfsonlinux/spl/issues/218

I had missed this when looking through the bug tracker. Thanks for bringing it to my attention. I will see what I can do to fix the build failure.

[ryao]

I tried building a 3.8.4 realtime kernel and was promptly greeted by a build failure for the kernel source tree itself:

Code: Select all

*/usr/src/linux/kernel/events/uprobes.c:1143: undefined reference to `percpu_down_read'
*kernel/built-in.o: In function `uprobe_end_dup_mmap':
*/usr/src/linux/kernel/events/uprobes.c:1148: undefined reference to `percpu_up_read'
*kernel/built-in.o: In function `init_uprobes':
*/usr/src/linux/kernel/events/uprobes.c:1617: undefined reference to `__percpu_init_rwsem'

I cannot produce a fix for ZFS unless I have a fix for this and I do not have time to fix issues in other people's kernel code right now. :/

[kernelOfTruth]

I tried building a 3.8.4 realtime kernel and was promptly greeted by a build failure for the kernel source tree itself:

Code: Select all

*/usr/src/linux/kernel/events/uprobes.c:1143: undefined reference to `percpu_down_read'
*kernel/built-in.o: In function `uprobe_end_dup_mmap':
*/usr/src/linux/kernel/events/uprobes.c:1148: undefined reference to `percpu_up_read'
*kernel/built-in.o: In function `init_uprobes':
*/usr/src/linux/kernel/events/uprobes.c:1617: undefined reference to `__percpu_init_rwsem'

I cannot produce a fix for ZFS unless I have a fix for this and I do not have time to fix issues in other people's kernel code right now. :/

I'll have a look at it & see whether I get it to build reliably within the next days

thanks again



on-topic:

using the latency-timer tweaks + BFS and I couldn't observe any sound stuttering so far

even while compiling GCC which ALWAYS had resulted in at least some minor stuttering or lag on the GUI & sound



I also followed advice and use schedtool on X & pulseaudio

Code: Select all

schedtool -p 0 -n -20 -N `pgrep X`

schedtool -p 0 -n -20 -N `pgrep pulseaudio`

[kernelOfTruth]

@ryao:

let's break out the discussion related to ZFSonLinux and continue in the following thread:

http://forums.gentoo.org/viewtopic-p-7276174.html

[kernelOfTruth]

the following RCU settings seem to improve things:

Code: Select all

zcat /proc/config.gz | grep RCU
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_USER_QS=y
CONFIG_RCU_FANOUT=32
CONFIG_RCU_FANOUT_LEAF=2
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=99
CONFIG_RCU_BOOST_DELAY=331
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_ALL=y
# RCU Debugging
# CONFIG_PROVE_RCU_DELAY is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=32
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
# CONFIG_RCU_TRACE is not set

also for those who often have heavy i/o going on on their system

the ROW i/o scheduler (read over write) might be interesting:

http://marc.info/?l=linux-mmc&m=137077433718450&w=2
http://marc.info/?l=linux-kernel&m=137077436518458&w=2
http://marc.info/?l=linux-kernel&m=137077469818542&w=2
http://marc.info/?l=linux-mmc&m=137077472018547&w=2

to cut down latencies

it currently seems to undergo some refinements but it's already widely used in the smartphone community XDA (e.g. I'm also using it on my Galaxy Note and it's working fine)



I've some more tweaks related to latency used in my system - when they've been extensively tested I'll post them in a sort of round-up

[yoshi314]

i'm going to try this.

So far the only major issue i have is when copying big files (1.5-2GB) to usb. probably due to memory pressure or something, everything slows down to a crawl until the file is copied over, especially during final 10% of the copy process (when using mc or bar application).

That's on 64bit system with core2 cpu and 3gb of ram. It hits pretty hard with ck-3.9 kernel, with no further manual tweaks.

[ulenrich]

So far the only major issue i have is when copying big files (1.5-2GB) to usb. probably due to memory pressure or something, everything slows down

Yes, for me using mc too. I suspect caching. Why the hell we need normal caching?

We need a special purpose cache for copying:
read-cache-write-forget

No fighting over obsolete data wants to keep staying in cache ...
Or is it just the old midnight source code missing to use an existing special Linux kernel function?