rcu_preempt detected stalls on CPUs/tasks

[georgia_tech_swagger]

Hello. I recently built a new system based off Haswell-E. These are the new high end Core i7 parts with 6 to 8 cores and 12 to 16 threads. I have had a hell of time with it slowly locking up over time. I am getting tons of kernel stall messages from kernel 3.16.x and 3.17.x and 3.18.7 right now. Some misc details and observations:

- I am running two GeForce 760 GTX cards. Only one is used though. I intend to use the other ultimately for KVM gaming of Windows only games. The second GPU shows up in nvidia-settings, but just with no monitor or anything attached. Which is what I'd expect at this time.

- Without a pre-emptable kernel ..... I'll get a week or two before it start getting the RCU vomit in dmesg. With a pre-emptable kernel ... a day.

- The symptoms are X applications stop rendering changes. Particularly JDownloader2, gkrellm, and any video output from mplayer. JDownloader stops working entirely. Gkrellm still works but stops all refreshing. And mplayer will play maybe a half second of video and then stop dead in its tracks. Given long enough ... all of X will simply lock up.

- It seems to be most readily caused by having mplayer and/or mplayer2 running shuffling through a stack of videos that runs for hours. This happens whether it runs using vdpau or xv.


Almost all of the dmesg errors look like this:

[97435.030056] INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 6, t=60002 jiffies, g=5123149, c=5123148, q=106034)
[97435.030056] Task dump for CPU 1:
[97435.030056] swapper/1 R running task 0 0 1 0x00200000
[97435.030056] ffff88043da57e98 ffffffff81465479 0000000000000293 ffff88043da57fd8
[97435.030056] ffff88043d9fe7f0 00000000000104c0 ffff88033104b7f0 ffff88033104b7f0
[97435.030056] 0000000000000000 ffff88043da57fd8 ffff88043da57e98 ffff88043da57fd8
[97435.030056] Call Trace:
[97435.030056] [<ffffffff81465479>] ? __schedule+0x679/0x831
[97435.030056] [<ffffffff81221b70>] ? debug_smp_processor_id+0x17/0x19
[97435.030056] [<ffffffff8127c045>] ? intel_idle+0x45/0x113
[97435.030056] [<ffffffff8137119b>] ? cpuidle_enter_state+0x3a/0x93
[97435.030056] [<ffffffff81371293>] ? cpuidle_enter+0x12/0x14
[97435.030056] [<ffffffff8105b563>] ? cpu_startup_entry+0x19b/0x265
[97435.030056] [<ffffffff81024eee>] ? start_secondary+0x120/0x13e



But once (and only once!) I caught this message:

[96459.447019] INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 0, t=60002 jiffies, g=5121270, c=5121269, q=61653)
[96459.447036] Task dump for CPU 1:
[96459.447037] opera R running task 0 16108 15908 0x00000000
[96459.447037] ffff880386ddfa48 ffffffff81465479 ffff880386ddfa28 ffff880386ddffd8
[96459.447037] ffff88042eae67f0 00000000000104c0 ffff88043d9fe7f0 ffff88043d9fe7f0
[96459.447037] 0000000000000001 ffffffff813bb2d6 ffff880300000000 ffff88043fffa808
[96459.447037] Call Trace:
[96459.447037] [<ffffffff81465479>] ? __schedule+0x679/0x831
[96459.447037] [<ffffffff813bb2d6>] ? sch_direct_xmit+0xc4/0x1c3
[96459.447037] [<ffffffff8104f73f>] ? get_parent_ip+0xe/0x3e
[96459.447037] [<ffffffff8138d31d>] ? dlci_ioctl_set+0x2c/0x2c
[96459.447037] [<ffffffff8146569a>] schedule+0x69/0x6b
[96459.447037] [<ffffffff814687aa>] schedule_hrtimeout_range_clock+0x37/0xdb
[96459.447037] [<ffffffff8104f73f>] ? get_parent_ip+0xe/0x3e
[96459.447037] [<ffffffff81221b85>] ? __this_cpu_preempt_check+0x13/0x15
[96459.447037] [<ffffffff810b4cf4>] __inc_zone_state+0x40/0x8d
[96459.447037] [<ffffffff8104f73f>] ? get_parent_ip+0xe/0x3e
[96459.447037] [<ffffffff8104f7e5>] ? preempt_count_add+0x76/0x8a
[96459.447037] [<ffffffff810a7a9b>] ? get_page_from_freelist+0x63b/0x789
[96459.447037] [<ffffffff810a7d39>] ? __alloc_pages_nodemask+0x150/0x6e2
[96459.447037] [<ffffffff81221b85>] ? __this_cpu_preempt_check+0x13/0x15
[96459.447037] [<ffffffff81221b70>] ? debug_smp_processor_id+0x17/0x19
[96459.447037] [<ffffffff810ab7aa>] ? __lru_cache_add+0x6d/0x83
[96459.447037] [<ffffffff810abae3>] ? lru_cache_add_active_or_unevictable+0x23/0x70
[96459.447037] [<ffffffff81468cc1>] ? _raw_spin_unlock+0x11/0x24
[96459.447037] [<ffffffff810bea83>] ? handle_mm_fault+0x3c2/0xbf1
[96459.447037] [<ffffffff8102dde0>] ? __do_page_fault+0x31d/0x3fb
[96459.447037] [<ffffffff810b39d0>] ? vm_mmap_pgoff+0x6d/0x8f
[96459.447037] [<ffffffff8102deeb>] ? do_page_fault+0xc/0xe
[96459.447037] [<ffffffff8146aa72>] ? page_fault+0x22/0x30



When mplayer does lock up and I Ctrl+C it, I get this:

MPlayer interrupted by signal 2 in module: sleep_timer
A: 0.3 V: 0.2 A-V: 0.175 ct: -0.123 6/ 6 ??% ??% ??,?% 0 0



So ... my best guess .... is this a kernel bug with the cpuidle timer?? This seems like a really strange / slow to develop thing to be a CPU fault?



# eix xorg-server
[I] x11-base/xorg-server
Available versions: 1.12.4-r4(0/1.12.4) 1.15.2-r2(0/1.15.2) 1.16.4(0/1.16.1) ~1.16.4-r1(0/1.16.1) ~1.16.4-r2(0/1.16.1) ~1.17.1(0/1.17.1) ~1.17.1-r1(0/1.17.1) {dmx doc glamor ipv6 kdrive minimal nptl selinux static-libs +suid systemd tslib +udev unwind wayland xephyr xnest xorg xvfb}
Installed versions: 1.16.4(08:39:48 03/10/15)(glamor ipv6 nptl suid udev xorg xvfb -dmx -doc -kdrive -minimal -selinux -static-libs -systemd -tslib -unwind -wayland -xnest)
Homepage: http://xorg.freedesktop.org/
Description: X.Org X servers

# eix nvidia-drivers
[I] x11-drivers/nvidia-drivers
Available versions: [M]96.43.23^msd [M]173.14.39^msd 304.125^msd 331.113^msd 340.76^msd 343.36^msd 346.47^msd {+X acpi custom-cflags gtk gtk2 gtk3 multilib pax_kernel (+)tools uvm KERNEL="FreeBSD linux"}
Installed versions: 346.47^msd(09:00:19 03/17/15)(X acpi gtk2 gtk3 multilib tools -pax_kernel -uvm KERNEL="linux -FreeBSD")
Homepage: http://www.nvidia.com/
Description: NVIDIA Accelerated Graphics Driver

# uname -r
3.18.7-gentoo


My GRUB line:

title Gentoo Linux 3.18.7
root (hd0,0)
kernel /boot/kernel-genkernel-x86_64-3.18.7-gentoo root=/dev/sda1 rootfstype=ext4 acpi_enforce_resources=lax vga=792
initrd /boot/initramfs-genkernel-x86_64-3.18.7-gentoo



I am running with the acpi lax bit because my motherboard's temperature sensor is undetectable without that. I tried on 3.16.x and 3.17.x booting without that param to no real difference.

[kernelOfTruth]

Hi georgia_tech_swagger,


please use the code & quote tags to improve readability