| View previous topic :: View next topic |
| Author |
Message |
rubik-wuerfel
n00b
Joined: 29 Sep 2004
Posts: 53
Location: Goettingen, Germany
|
 Posted: Sat Oct 03, 2015 7:45 pm Post subject: System freeze related to "rcu_sched detected stalls on |
 |
|
Hello,
I just installed Gentoo on this new machine, and experience a number of system freezes, which seem to be correlated to this output, as recorded by journalctl:
| Code: | Oct 03 20:36:55 droggelbecher kernel: INFO: rcu_sched detected stalls on CPUs/tasks:
Oct 03 20:36:55 droggelbecher kernel: 3: (1 GPs behind) idle=5af/1/0 softirq=71484/71485 fqs=2098
Oct 03 20:36:55 droggelbecher kernel: (detected by 1, t=2102 jiffies, g=61577, c=61576, q=9555)
Oct 03 20:36:55 droggelbecher kernel: Task dump for CPU 3:
Oct 03 20:36:55 droggelbecher kernel: swapper/3 R running task 0 0 1 0x00000008
Oct 03 20:36:55 droggelbecher kernel: ffffffffffffff4c ffffffff814dd7d8 0000000000000010 0000000000000212
Oct 03 20:36:55 droggelbecher kernel: ffff88021d963e88 0000000000000018 ffffffff814dd7d4 000000021d960000
Oct 03 20:36:55 droggelbecher kernel: ffffffffa089efb0 ffffffffa089efb0 ffff88021278e000 ffffffffa089efb0
Oct 03 20:36:55 droggelbecher kernel: Call Trace:
Oct 03 20:36:55 droggelbecher kernel: [<ffffffff814dd7d8>] ? cpuidle_enter_state+0x46/0x93
Oct 03 20:36:55 droggelbecher kernel: [<ffffffff814dd7d4>] ? cpuidle_enter_state+0x42/0x93
Oct 03 20:36:55 droggelbecher kernel: [<ffffffff814dd847>] ? cpuidle_enter+0x12/0x14
Oct 03 20:36:55 droggelbecher kernel: [<ffffffff81077f6a>] ? cpu_startup_entry+0x1c0/0x2a9
Oct 03 20:36:55 droggelbecher kernel: [<ffffffff81024fbe>] ? start_secondary+0x115/0x133
Oct 03 20:37:31 droggelbecher systemd[1]: systemd-logind.service watchdog timeout (limit 1min)!
Oct 03 20:37:49 droggelbecher kernel: sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m
Oct 03 20:37:53 droggelbecher kernel: sysrq: SysRq : SAK
Oct 03 20:37:53 droggelbecher kernel: SAK: killed process 2125 (X): task_session(p)==tty->session
Oct 03 20:37:53 droggelbecher kernel: SAK: killed process 2125 (X): task_session(p)==tty->session
Oct 03 20:37:55 droggelbecher kernel: sysrq: SysRq : Emergency Sync |
In this specific case I could not even switch from X11 to the console, fortunately the MagicSysReq keys still worked (on some occasions, even this route for rebooting was blocked).
Any idea what might cause these freezes? (X becomes noticeably slower for a few seconds, then I cannot even move the mouse any more.)
Possibly relevant info:
| Code: | # lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5
01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 720] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
02:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) |
| Code: | # emerge --info
Portage 2.2.20.1 (python 3.4.3-final-0, default/linux/amd64/13.0/desktop/kde/systemd, gcc-4.8.5, glibc-2.20-r2, 4.0.5-gentoo x86_64)
=================================================================
System uname: Linux-4.0.5-gentoo-x86_64-AMD_FX-tm-4300_Quad-Core_Processor-with-gentoo-2.2
KiB Mem: 8131008 total, 6148164 free
KiB Swap: 40569852 total, 40569852 free
Timestamp of repository gentoo: Thu, 01 Oct 2015 19:45:01 +0000
sh bash 4.3_p39
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash: 4.3_p39::gentoo
dev-lang/perl: 5.20.2::gentoo
dev-lang/python: 2.7.10::gentoo, 3.4.3::gentoo
dev-util/cmake: 3.2.2::gentoo
dev-util/pkgconfig: 0.28-r2::gentoo
sys-apps/baselayout: 2.2::gentoo
sys-apps/openrc: 0.17::gentoo
sys-apps/sandbox: 2.6-r1::gentoo
sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo
sys-devel/automake: 1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils: 2.25.1-r1::gentoo
sys-devel/gcc: 4.8.5::gentoo, 4.9.3::gentoo
sys-devel/gcc-config: 1.7.3::gentoo
sys-devel/libtool: 2.4.6::gentoo
sys-devel/make: 4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc: 2.20-r2::gentoo
Repositories:
gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.de.gentoo.org/gentoo-portage
priority: -1000
sakaki-tools
location: /usr/local/portage/sakaki-tools
sync-type: git
sync-uri: https://github.com/sakaki-/sakaki-tools.git
masters: gentoo
priority: 50
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--ask --jobs=5 --load-average=5"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://mirror.leaseweb.com/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5 -l5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi aes alsa amd64 avx bash-completion berkdb bindist bluetooth branding bzip2 cairo cdda cdr cli cracklib crypt cryptsetup cups cxx dbus declarative dri dts dvd dvdr emacs emboss encode exif fam ffmpeg firefox flac fma3 fma4 fortran gdbm gif glamor gpm gtk gtk3 iconv ipv6 jpeg kde kipi lcms ldap libav libnotify lm_sensors mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp otr pam pango pcre pdf phonon plasma png policykit popcnt ppds python qt3support qt4 qt5 readline sdl seccomp semantic-desktop session sound spell sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vorbis wxwidgets x264 xcb xcomposite xinerama xml xmpp xop xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx fma3 fma4 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="nvidia nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
|
|
|
| Back to top |
|
 |
rubik-wuerfel
n00b
Joined: 29 Sep 2004
Posts: 53
Location: Goettingen, Germany
|
| Posted: Sat Oct 03, 2015 7:55 pm Post subject: |
|
|
Two more remarks:
- grepping through the old rcu_sched messages shows that the task dump was always for CPU 3. Could this be relevant or is this behavior to be expected?
- I don't know if this is real or just my imagination, but I think that most (but definitely not all) of these freezes happened while browsing the internet, maybe even mostly while entering search terms in the search bar of Firefox.
|
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Tue Oct 13, 2015 9:07 pm Post subject: |
|
|
Have you continued to experience this issue?
I have a HP Proliant DL380 G7 that just started doing this today. I do not have X installed, or any such things - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.
The server has been deployed for about 5 months. Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.
I ended up not using VT-d for other reasons, but left intel_iommu=on in place.
Today the server locked up with
| Quote: | Oct 13 14:58:38 c2 kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=8408 jiffies, g=13505, c=13504, q=348)
Oct 13 14:58:38 c2 kernel: Task dump for CPU 0:
Oct 13 14:58:38 c2 kernel: swapper/0 R running task 0 0 0 0x00000008
Oct 13 14:58:38 c2 kernel: ffffffffffffff10 ffffffffb89a829a 0000000000000010 0000000000000216
Oct 13 14:58:38 c2 kernel: ffffffffb8c33f28 0000000000000018 ffffffffb89a828c ffffffffb8c9fbb8
Oct 13 14:58:38 c2 kernel: ffff8811dfc1aa70 ffffffffb8c76e80 ffffffffb8c30000 ffffffffb8c30000
Oct 13 14:58:38 c2 kernel: Call Trace:
Oct 13 14:58:38 c2 kernel: [<ffffffffb89a829a>] ? cpuidle_enter_state+0x9a/0x140
Oct 13 14:58:38 c2 kernel: [<ffffffffb89a828c>] ? cpuidle_enter_state+0x8c/0x140
Oct 13 14:58:38 c2 kernel: [<ffffffffb86a3daf>] ? cpu_startup_entry+0x23f/0x2d0
Oct 13 14:58:38 c2 kernel: [<ffffffffb8cbee7e>] ? start_kernel+0x370/0x37b
Oct 13 14:58:38 c2 kernel: [<ffffffffb8cbe919>] ? set_init_arg+0x4e/0x4e
|
I rebooted with intel_iommu=on removed from kernel boot, it ran for about 30 minutes then died with the same exact message.
I then rebooted & turned VT-d off in the BIOS...I have now been running for about 40 minutes.
Mine also had Task dump for CPU 0 on all messages.
I'm not sure how IOMMU relates to the RCU system - do you have IOMMU turned on, or VT-d enabled in your BIOS?
The server was running vanilla kernel 4.1.6 since deployment, but I updated to 4.1.10 when I turned intel_iommu=on. What kernel are you using? |
|
| Back to top |
|
|
rubik-wuerfel
n00b
Joined: 29 Sep 2004
Posts: 53
Location: Goettingen, Germany
|
| Posted: Fri Oct 16, 2015 7:16 pm Post subject: |
|
|
This looks clearly as if there are different reasons.
| rickvernam wrote: | | I do not have X installed, or any such things |
Same here, it started on a fresh install, even before I had started to emerge X.
| rickvernam wrote: | - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.
The server has been deployed for about 5 months. Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.
...
I'm not sure how IOMMU relates to the RCU system - do you have IOMMU turned on, or VT-d enabled in your BIOS? |
AMD cpu here, problems with both 4.0.5 and 4.2.3, and no virtualization processes in place.
On the other hand, I was able to get rid of the freezes by switching from the open source nvidia gfx drivers (nvidiafb & nouveau) to the proprietary nvidia drivers. Because the problems first appeared without X, I believe that probably nvidiafb is somehow involved. |
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Fri Oct 16, 2015 9:12 pm Post subject: |
|
|
| Just to follow up for any future readers, I also have not had an occurrence of this since I removed intel_iommu=on line from kernel boot command line and turned VT-d back off in the BIOS. |
|
| Back to top |
|
|
Roman_Gruber
Advocate
Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria
|
| Posted: Sat Oct 17, 2015 12:39 am Post subject: |
|
|
| microcode is loaded? |
|
| Back to top |
|
|
rubik-wuerfel
n00b
Joined: 29 Sep 2004
Posts: 53
Location: Goettingen, Germany
|
| Posted: Sat Oct 17, 2015 7:41 am Post subject: |
|
|
The microcode module is loaded, sys-kernel/linux-firmware is installed, excerpt from my .config:
| Code: | /usr/src/linux # grep MICROCODE .config
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y |
|
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Sat Oct 17, 2015 1:11 pm Post subject: |
|
|
| tw04l124 wrote: | | microcode is loaded? |
I believe so:
| Code: | c2 linux # grep MICROCODE .config
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
# CONFIG_MICROCODE_AMD is not set
CONFIG_MICROCODE_OLD_INTERFACE=y
|
and
| Code: | c2 linux # dmesg| grep microcode
[ 1.005308] microcode: CPU0 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005317] microcode: CPU1 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005327] microcode: CPU2 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005336] microcode: CPU3 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005345] microcode: CPU4 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005354] microcode: CPU5 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005364] microcode: CPU6 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005374] microcode: CPU7 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005384] microcode: CPU8 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005393] microcode: CPU9 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005403] microcode: CPU10 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005412] microcode: CPU11 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005421] microcode: CPU12 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005430] microcode: CPU13 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005439] microcode: CPU14 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005447] microcode: CPU15 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005455] microcode: CPU16 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005463] microcode: CPU17 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005472] microcode: CPU18 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005480] microcode: CPU19 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005489] microcode: CPU20 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005496] microcode: CPU21 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005505] microcode: CPU22 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005512] microcode: CPU23 sig=0x206c2, pf=0x1, revision=0x1a
[ 1.005567] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
|
It appears I can install microcode-ctl & intel-microcode. This is a production server, so I have to wait until their shift is over before I can do this.
I also had this reoccur today, alternating between
| Code: | Oct 17 04:10:05 c2 kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 12, t=102982 jiffies, g=139, c=138, q=8)
Oct 17 04:10:05 c2 kernel: Task dump for CPU 0:
Oct 17 04:10:05 c2 kernel: swapper/0 R running task 0 0 0 0x00000808
Oct 17 04:10:05 c2 kernel: ffffffffffffff10 ffffffff821a829a 0000000000000010 0000000000000202
Oct 17 04:10:05 c2 kernel: ffffffff82433f28 0000000000000018 ffffffff821a828c ffffffff8249fbb8
Oct 17 04:10:05 c2 kernel: ffff8811dfc1aa70 ffffffff82476e80 ffffffff82430000 ffffffff82430000
Oct 17 04:10:05 c2 kernel: Call Trace:
Oct 17 04:10:05 c2 kernel: [<ffffffff821a829a>] ? cpuidle_enter_state+0x9a/0x140
Oct 17 04:10:05 c2 kernel: [<ffffffff821a828c>] ? cpuidle_enter_state+0x8c/0x140
Oct 17 04:10:05 c2 kernel: [<ffffffff81ea3daf>] ? cpu_startup_entry+0x23f/0x2d0
Oct 17 04:10:05 c2 kernel: [<ffffffff824bee7e>] ? start_kernel+0x370/0x37b
Oct 17 04:10:05 c2 kernel: [<ffffffff824be919>] ? set_init_arg+0x4e/0x4e
|
and
| Code: | Oct 17 04:10:13 c2 kernel: INFO: rcu_sched self-detected stall on CPU { 0} (t=134463 jiffies g=887585 c=887584 q=12053)
Oct 17 04:10:13 c2 kernel: Task dump for CPU 0:
Oct 17 04:10:13 c2 kernel: swapper/0 R running task 0 0 0 0x00000808
Oct 17 04:10:13 c2 kernel: 0000000000000001 ffffffff824674c0 ffffffff81eb1b5f 00000000000d8b21
Oct 17 04:10:13 c2 kernel: ffff8811dfc154c0 ffffffff82467400 ffffffff824674c0 ffffffff8249fbc0
Oct 17 04:10:13 c2 kernel: ffffffff81eb4bea 0000989680000000 0000000000002f15 0000000000000000
Oct 17 04:10:13 c2 kernel: Call Trace:
Oct 17 04:10:13 c2 kernel: <IRQ> [<ffffffff81eb1b5f>] ? rcu_dump_cpu_stacks+0x7f/0xc0
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb4bea>] ? rcu_check_callbacks+0x3ca/0x6f0
Oct 17 04:10:13 c2 kernel: [<ffffffff81ebbf51>] ? timekeeping_update.constprop.9+0xe1/0x120
Oct 17 04:10:13 c2 kernel: [<ffffffff81ebd136>] ? update_wall_time+0x236/0x660
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb6d01>] ? update_process_times+0x31/0x60
Oct 17 04:10:13 c2 kernel: [<ffffffff81ec4c23>] ? tick_sched_timer+0x33/0x60
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb755b>] ? __run_hrtimer.isra.34+0x3b/0xc0
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb7915>] ? hrtimer_interrupt+0xd5/0x210
Oct 17 04:10:13 c2 kernel: [<ffffffff81e30da5>] ? smp_apic_timer_interrupt+0x35/0x50
Oct 17 04:10:13 c2 kernel: [<ffffffff822a8c3b>] ? apic_timer_interrupt+0x6b/0x70
Oct 17 04:10:13 c2 kernel: [<ffffffff822a7975>] ? _raw_spin_unlock_irqrestore+0x5/0x10
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb632f>] ? try_to_del_timer_sync+0x3f/0x60
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb62f0>] ? del_timer+0x50/0x50
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb6392>] ? del_timer_sync+0x42/0x50
Oct 17 04:10:13 c2 kernel: [<ffffffff82209e4f>] ? inet_csk_reqsk_queue_drop+0x5f/0x1b0
Oct 17 04:10:13 c2 kernel: [<ffffffff8220a1f8>] ? reqsk_timer_handler+0x258/0x2c0
Oct 17 04:10:13 c2 kernel: [<ffffffff82209fa0>] ? inet_csk_reqsk_queue_drop+0x1b0/0x1b0
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb6495>] ? call_timer_fn.isra.31+0x15/0x70
Oct 17 04:10:13 c2 kernel: [<ffffffff81eb6692>] ? run_timer_softirq+0x1a2/0x230
Oct 17 04:10:13 c2 kernel: [<ffffffff81e76d44>] ? __do_softirq+0xd4/0x1e0
Oct 17 04:10:13 c2 kernel: [<ffffffff81e77045>] ? irq_exit+0x95/0xa0
Oct 17 04:10:13 c2 kernel: [<ffffffff81e30daa>] ? smp_apic_timer_interrupt+0x3a/0x50
Oct 17 04:10:13 c2 kernel: [<ffffffff822a8c3b>] ? apic_timer_interrupt+0x6b/0x70
Oct 17 04:10:13 c2 kernel: <EOI> [<ffffffff821a829a>] ? cpuidle_enter_state+0x9a/0x140
Oct 17 04:10:13 c2 kernel: [<ffffffff821a828c>] ? cpuidle_enter_state+0x8c/0x140
Oct 17 04:10:13 c2 kernel: [<ffffffff81ea3daf>] ? cpu_startup_entry+0x23f/0x2d0
Oct 17 04:10:13 c2 kernel: [<ffffffff824bee7e>] ? start_kernel+0x370/0x37b
Oct 17 04:10:13 c2 kernel: [<ffffffff824be919>] ? set_init_arg+0x4e/0x4e
|
for about 20 minutes before it locked hard. It's always CPU 0, still. |
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Sat Oct 17, 2015 2:03 pm Post subject: |
|
|
| rickvernam wrote: | | tw04l124 wrote: | | microcode is loaded? |
...
It appears I can install microcode-ctl & intel-microcode. This is a production server, so I have to wait until their shift is over before I can do this.
... |
Do I understand from this correctly that there is not a microcode update available?
| Code: | c2 ~ # iucode_tool -S --write-earlyfw=/boot/early-ucode.cpio /lib/firmware/intel-ucode/*
iucode_tool: system has processor(s) with signature 0x000206c2
iucode_tool: No valid microcodes were selected, nothing to do...
|
My processor is
| Code: | c2 ~ # head -n10 /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz
stepping : 2
microcode : 0x1a
|
So the microcode file should be 06-2C-02 I suppose, but there is no such a file in /lib/firmware/intel-ucode.
Is there something else I can be doing regarding microcode?
Thanks! |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Sat Oct 17, 2015 8:19 pm Post subject: |
|
|
| rickvernam wrote: | Have you continued to experience this issue?
I have a HP Proliant DL380 G7 that just started doing this today. I do not have X installed, or any such things - just base system + apache, postgres, KMV / libvirt, and a few less interesting things.
The server has been deployed for about 5 months. Two days ago I rebooted with intel_iommu=on for the first time so that I could make use of VT-d to get some hardware into a KVM instance.
I ended up not using VT-d for other reasons, but left intel_iommu=on in place.
Today the server locked up with
| Quote: | Oct 13 14:58:38 c2 kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=8408 jiffies, g=13505, c=13504, q=348)
Oct 13 14:58:38 c2 kernel: Task dump for CPU 0:
Oct 13 14:58:38 c2 kernel: swapper/0 R running task 0 0 0 0x00000008
Oct 13 14:58:38 c2 kernel: ffffffffffffff10 ffffffffb89a829a 0000000000000010 0000000000000216
Oct 13 14:58:38 c2 kernel: ffffffffb8c33f28 0000000000000018 ffffffffb89a828c ffffffffb8c9fbb8
Oct 13 14:58:38 c2 kernel: ffff8811dfc1aa70 ffffffffb8c76e80 ffffffffb8c30000 ffffffffb8c30000
Oct 13 14:58:38 c2 kernel: Call Trace:
Oct 13 14:58:38 c2 kernel: [<ffffffffb89a829a>] ? cpuidle_enter_state+0x9a/0x140
Oct 13 14:58:38 c2 kernel: [<ffffffffb89a828c>] ? cpuidle_enter_state+0x8c/0x140
Oct 13 14:58:38 c2 kernel: [<ffffffffb86a3daf>] ? cpu_startup_entry+0x23f/0x2d0
Oct 13 14:58:38 c2 kernel: [<ffffffffb8cbee7e>] ? start_kernel+0x370/0x37b
Oct 13 14:58:38 c2 kernel: [<ffffffffb8cbe919>] ? set_init_arg+0x4e/0x4e
|
I rebooted with intel_iommu=on removed from kernel boot, it ran for about 30 minutes then died with the same exact message.
I then rebooted & turned VT-d off in the BIOS...I have now been running for about 40 minutes.
Mine also had Task dump for CPU 0 on all messages.
I'm not sure how IOMMU relates to the RCU system - do you have IOMMU turned on, or VT-d enabled in your BIOS?
The server was running vanilla kernel 4.1.6 since deployment, but I updated to 4.1.10 when I turned intel_iommu=on. What kernel are you using? |
It's looks like intel X58 errata hardware bug but i didn't find x58!?
http://thr3ads.net/xen-devel/2013/03/2286949-PATCH-0-3-IOMMU-errata-treatment-adjustments
Just look at [PATCH 3/3]
This article contains information about the Intel 55x0 chipset errata - Interrupt remapping issue. Intel 5500/5520/X58 chipset revision 0x13 and 0x22 have an errata (#47 and #53) which makes the IOMMU interrupt remapping unit unreliable. This erratum causes interruptions and the interrupt remapping invalidations become unresponsive.
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Mon Oct 19, 2015 4:57 pm Post subject: |
|
|
| TigerJr wrote: |
It's looks like intel X58 errata hardware bug but i didn't find x58!?
http://thr3ads.net/xen-devel/2013/03/2286949-PATCH-0-3-IOMMU-errata-treatment-adjustments
Just look at [PATCH 3/3]
This article contains information about the Intel 55x0 chipset errata - Interrupt remapping issue. Intel 5500/5520/X58 chipset revision 0x13 and 0x22 have an errata (#47 and #53) which makes the IOMMU interrupt remapping unit unreliable. This erratum causes interruptions and the interrupt remapping invalidations become unresponsive. |
The link references stepping levels, but I do not know how to relate the stepping information from that link. my /proc/cpuinfo says "stepping: 2", but the link says things like stepping B-2 & C-3. Where does that letter come from, how do I know which I have?
Anyway, I have VT-d turned off in the BIOS, though I do not believe there is an option to turn interrupt remapping on/off.
If I am affected by that errata, would the following be sufficient to avoid hitting the interrupt remapping issue and triggering the lock-up?
| Code: | --- IOMMU Hardware Support
Generic IOMMU Pagetable Support --->
[ ] AMD IOMMU support
[*] Support for Intel IOMMU using DMA Remapping Devices
[*] Enable Intel DMA Remapping Devices by default (NEW)
[ ] Support for Interrupt Remapping |
I was booted into a kernel that had Support for Interrupt Remapping turned on when I would hit this lock-up...
Thanks for your insight! |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Tue Oct 20, 2015 10:44 am Post subject: |
|
|
Interupt remmaping not always used by AMD VT-d , but if turn it off you will never use USB\PCI Passthrough.
| Quote: | | The link references stepping levels, but I do not know how to relate the stepping information from that link. my /proc/cpuinfo says "stepping: 2", but the link says things like stepping B-2 & C-3. Where does that letter come from, how do I know which I have? |
It's not a cpu stepping, it's revision of southbridge chipset use for that information dmidecode.
But lspci -k can also give you revision of your devices.
And than you post your lspci output i didn't see i5500 or i5520 or x58 chipsets those can have revisions(B2 C3).
About IOMMU, in lspci your mainboard have device:
| Code: | | 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) |
If you watch lspci -k output you can see what Kernel driver is in use: amd_iommu
But if your kernel compiled without that driver than I\O Virtualisation would be disabled. It's a part of AMD VT-d.
Here can hide hardware bug with interrupt remapping or it's just kernel not configured properly.
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Tue Oct 20, 2015 1:40 pm Post subject: |
|
|
Okay, so I do have hardware affected by the errata which you described, rev 0x13.
A section of my lspci: | Code: | 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)
00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13) |
So I will leave Interrupt Remapping turned off in the kernel. That may disable VT-d, but I can live with that.
| Code: | --- IOMMU Hardware Support
Generic IOMMU Pagetable Support --->
[ ] AMD IOMMU support
[*] Support for Intel IOMMU using DMA Remapping Devices
[*] Enable Intel DMA Remapping Devices by default (NEW)
[ ] Support for Interrupt Remapping |
Thanks again for your input & direction! |
|
| Back to top |
|
|
Polyatomic
n00b
Joined: 18 May 2014
Posts: 36
|
| Posted: Wed Oct 21, 2015 10:36 pm Post subject: |
|
|
Brain damage removed
Sat Oct 31 21:08:42 ACDT 2015
Last edited by Polyatomic on Sat Oct 31, 2015 10:40 am; edited 1 time in total |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
|
| Back to top |
|
|
Polyatomic
n00b
Joined: 18 May 2014
Posts: 36
|
| Posted: Sun Oct 25, 2015 12:47 am Post subject: |
|
|
Brain damage removed
Sat Oct 31 21:09:33 ACDT 2015
Last edited by Polyatomic on Sat Oct 31, 2015 10:41 am; edited 3 times in total |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Sun Oct 25, 2015 10:11 am Post subject: |
|
|
Can you post here your lspci output for revision of your chipset?
And i will give you another clue, errata is not a one problem that people have faced using vt-d.
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
Polyatomic
n00b
Joined: 18 May 2014
Posts: 36
|
| Posted: Sun Oct 25, 2015 8:43 pm Post subject: |
|
|
| Reserved! |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Sun Oct 25, 2015 9:42 pm Post subject: |
|
|
For selling? =) You solve this problem!
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Sun Oct 25, 2015 11:24 pm Post subject: |
|
|
So i have faced with this bug too, on these hardware:
| Quote: | localhost linux # lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Device 1234:1111
00:03.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
|
It's xen guest and dmesg output interesting too:
| Code: | localhost linux # dmesg
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.2.4 (root@next.xenogentoo.org) (gcc version 4.9.3 (Gentoo 4.9.3 p1.3, pie-0.6.3) ) #1 SMP Sun Oct 25 19:04:00 2015
[ 0.000000] Command line: root=/dev/xvda1
[ 0.000000] tseg: 0000000000
[ 0.000000] x86/fpu: Legacy x87 FPU detected.
[ 0.000000] x86/fpu: Using 'lazy' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007f7fffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.4 present.
[ 0.000000] DMI: Xen HVM domU, BIOS 4.6.0 10/18/2015
[ 0.000000] Hypervisor detected: Xen
[ 0.000000] Xen version 4.6.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
You might have to change the root device
from /dev/hd[a-d] to /dev/xvd[a-d]
in your root= kernel command line option
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x7f800 max_arch_pfn = 0x400000000
[ 0.000000] MTRR default type: write-back
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-BFFFF write-combining
[ 0.000000] C0000-FFFFF write-back
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 00F0000000 mask FFF0000000 uncachable
[ 0.000000] 1 disabled
[ 0.000000] 2 disabled
[ 0.000000] 3 disabled
[ 0.000000] 4 disabled
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] TOM2: 0000000200000000 aka 8192M
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT
[ 0.000000] found SMP MP-table at [mem 0x000fb710-0x000fb71f] mapped at [ffff8800000fb710]
[ 0.000000] Base memory trampoline at [ffff880000098000] 98000 size 24576
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[ 0.000000] [mem 0x00000000-0x000fffff] page 4k
[ 0.000000] BRK [0x01b03000, 0x01b03fff] PGTABLE
[ 0.000000] BRK [0x01b04000, 0x01b04fff] PGTABLE
[ 0.000000] BRK [0x01b05000, 0x01b05fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x7f600000-0x7f7fffff]
[ 0.000000] [mem 0x7f600000-0x7f7fffff] page 2M
[ 0.000000] BRK [0x01b06000, 0x01b06fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x60000000-0x7f5fffff]
[ 0.000000] [mem 0x60000000-0x7f5fffff] page 2M
[ 0.000000] init_memory_mapping: [mem 0x40000000-0x5fffffff]
[ 0.000000] [mem 0x40000000-0x5fffffff] page 2M
[ 0.000000] init_memory_mapping: [mem 0x00100000-0x3fffffff]
[ 0.000000] [mem 0x00100000-0x001fffff] page 4k
[ 0.000000] [mem 0x00200000-0x3fffffff] page 2M
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen )
[ 0.000000] ACPI: XSDT 0x00000000FC00D000 000054 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 0x00000000FC00CCC0 0000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 0x00000000FC0042E0 008956 (v02 Xen HVM 00000000 INTL 20140828)
[ 0.000000] ACPI: FACS 0x00000000FC0042A0 000040
[ 0.000000] ACPI: FACS 0x00000000FC0042A0 000040
[ 0.000000] ACPI: APIC 0x00000000FC00CDC0 0000D8 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 0x00000000FC00CF10 000038 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: WAET 0x00000000FC00CF50 000028 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: SSDT 0x00000000FC00CF80 000031 (v02 Xen HVM 00000000 INTL 20140828)
[ 0.000000] ACPI: SSDT 0x00000000FC00CFC0 000031 (v02 Xen HVM 00000000 INTL 20140828)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007f7fffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x7f7fd000-0x7f7fffff]
[ 0.000000] [ffffea0000000000-ffffea0001ffffff] PMD -> [ffff88007ce00000-ffff88007edfffff] on node 0
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007f7fffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009dfff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x000000007f7fffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000007f7fffff]
[ 0.000000] On node 0 totalpages: 522141
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3997 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 8096 pages used for memmap
[ 0.000000] DMA32 zone: 518144 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0xb008
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ5 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] ACPI: IRQ10 used by override.
[ 0.000000] ACPI: IRQ11 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.000000] smpboot: 15 Processors exceeds NR_CPUS limit of 8
[ 0.000000] smpboot: Allowing 8 CPUs, 6 hotplug CPUs
[ 0.000000] e820: [mem 0x7f800000-0xfbffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen HVM
[ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 31 pages/cpu @ffff88007f400000 s89368 r8192 d29416 u262144
[ 0.000000] pcpu-alloc: s89368 r8192 d29416 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7
[ 0.000000] xen: PV spinlocks enabled
[ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 513960
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: root=/dev/xvda1
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Calgary: detecting Calgary via BIOS EBDA area
[ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[ 0.000000] Memory: 2043152K/2088564K available (6759K kernel code, 639K rwdata, 2084K rodata, 1000K init, 740K bss, 45412K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] NR_IRQS:4352 nr_irqs:896 16
[ 0.000000] xen:events: Using FIFO-based ABI
[ 0.000000] xen:events: Xen HVM callback vector for event delivery is enabled
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[ 0.000000] hpet clockevent registered
[ 0.000000] tsc: Detected 3000.174 MHz processor
[ 0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[ 0.005000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6000.34 BogoMIPS (lpj=3000174)
[ 0.009012] pid_max: default: 32768 minimum: 301
[ 0.013029] ACPI: Core revision 20150619
[ 0.022288] ACPI: All ACPI Tables successfully acquired
[ 0.028909] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 0.034488] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.038619] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.042046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.045479] Initializing cgroup subsys freezer
[ 0.049205] CPU: Physical Processor ID: 0
[ 0.052011] CPU: Processor Core ID: 0
[ 0.055012] process: using AMD E400 aware idle routine
[ 0.059015] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4
[ 0.062011] Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4, 1GB 0
[ 0.068737] Freeing SMP alternatives memory: 28K (ffffffff81a41000 - ffffffff81a48000)
[ 0.085453] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.089000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[ 0.101982] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.105020] Xen: using vcpuop timer interface
[ 0.105035] installing Xen timer for CPU 0
[ 0.108080] smpboot: CPU0: Dual-Core AMD Opteron(tm) Processor 2222 (fam: 0f, model: 41, stepping: 03)
[ 0.110059] cpu 0 spinlock event irq 53
[ 0.111102] Performance Events: Broken PMU hardware detected, using software events only.
[ 0.112018] Failed to access perfctr msr (MSR c0010007 is 0)
[ 0.113868] installing Xen timer for CPU 1
[ 0.114093] x86: Booting SMP configuration:
[ 0.115018] .... node #0, CPUs: #1cpu 1 spinlock event irq 59
[ 0.178050] x86: Booted up 1 node, 2 CPUs
[ 0.179018] smpboot: Total of 2 processors activated (12083.59 BogoMIPS)
[ 0.180544] devtmpfs: initialized
[ 0.186158] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 0.190112] RTC time: 20:10:42, date: 10/25/15
[ 0.191177] NET: Registered protocol family 16
[ 0.197032] cpuidle: using governor ladder
[ 0.203021] cpuidle: using governor menu
[ 0.205234] ACPI: bus type PCI registered
[ 0.207019] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.210316] PCI: Using configuration type 1 for base access
[ 0.221089] ACPI: Added _OSI(Module Device)
[ 0.223059] ACPI: Added _OSI(Processor Device)
[ 0.225023] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.227019] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.230569] xen: --> pirq=16 -> irq=9 (gsi=9)
[ 0.234681] ACPI: Interpreter enabled
[ 0.237023] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150619/hwxface-580)
[ 0.241020] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150619/hwxface-580)
[ 0.246026] ACPI: (supports S0 S3 S5)
[ 0.248017] ACPI: Using IOAPIC for interrupt routing
[ 0.250057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.288488] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.290037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[ 0.292034] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[ 0.295125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ 0.299178] acpiphp: Slot [0] registered
[ 0.303015] acpiphp: Slot [3] registered
[ 0.306487] acpiphp: Slot [4] registered
[ 0.310628] acpiphp: Slot [5] registered
[ 0.314561] acpiphp: Slot [6] registered
[ 0.318564] acpiphp: Slot [7] registered
[ 0.322499] acpiphp: Slot [8] registered
[ 0.326024] acpiphp: Slot [9] registered
[ 0.329496] acpiphp: Slot [10] registered
[ 0.333017] acpiphp: Slot [11] registered
[ 0.337015] acpiphp: Slot [12] registered
[ 0.340491] acpiphp: Slot [13] registered
[ 0.344027] acpiphp: Slot [14] registered
[ 0.348015] acpiphp: Slot [15] registered
[ 0.351492] acpiphp: Slot [16] registered
[ 0.355015] acpiphp: Slot [17] registered
[ 0.358497] acpiphp: Slot [18] registered
[ 0.362500] acpiphp: Slot [19] registered
[ 0.366015] acpiphp: Slot [20] registered
[ 0.369494] acpiphp: Slot [21] registered
[ 0.373015] acpiphp: Slot [22] registered
[ 0.377023] acpiphp: Slot [23] registered
[ 0.380504] acpiphp: Slot [24] registered
[ 0.384502] acpiphp: Slot [25] registered
[ 0.388015] acpiphp: Slot [26] registered
[ 0.392003] acpiphp: Slot [27] registered
[ 0.395518] acpiphp: Slot [28] registered
[ 0.399047] acpiphp: Slot [29] registered
[ 0.461515] acpiphp: Slot [30] registered
[ 0.465065] acpiphp: Slot [31] registered
[ 0.468483] PCI host bridge to bus 0000:00
[ 0.472024] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.475020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.478019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.482021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 0.487019] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[ 0.493243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[ 0.496014] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[ 0.500023] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[ 0.501564] pci 0000:00:01.1: reg 0x20: [io 0xc200-0xc20f]
[ 0.502452] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
[ 0.506019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
[ 0.509019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
[ 0.513019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
[ 0.516808] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[ 0.516974] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
[ 0.527134] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
[ 0.534292] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000
[ 0.534924] pci 0000:00:02.0: reg 0x10: [mem 0xf1000000-0xf17fffff pref]
[ 0.538021] pci 0000:00:03.0: [5853:0001] type 00 class 0xff8000
[ 0.538410] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc0ff]
[ 0.538642] pci 0000:00:03.0: reg 0x14: [mem 0xf0000000-0xf0ffffff pref]
[ 0.542343] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
[ 0.546214] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.549207] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.553212] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
[ 0.565220] ACPI: Enabled 2 GPEs in block 00 to 0F
[ 0.568155] xen:balloon: Initialising balloon driver
[ 0.574242] vgaarb: setting as boot device: PCI:0000:00:02.0
[ 0.575000] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.580033] vgaarb: loaded
[ 0.583021] vgaarb: bridge control possible 0000:00:02.0
[ 0.585172] SCSI subsystem initialized
[ 0.587099] libata version 3.00 loaded.
[ 0.587133] ACPI: bus type USB registered
[ 0.589087] usbcore: registered new interface driver usbfs
[ 0.592053] usbcore: registered new interface driver hub
[ 0.594071] usbcore: registered new device driver usb
[ 0.596080] pps_core: LinuxPPS API ver. 1 registered
[ 0.598019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.602201] Advanced Linux Sound Architecture Driver Initialized.
[ 0.605088] PCI: Using ACPI for IRQ routing
[ 0.607021] PCI: pci_cache_line_size set to 64 bytes
[ 0.607394] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
[ 0.607400] e820: reserve RAM buffer [mem 0x7f800000-0x7fffffff]
[ 0.607681] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[ 0.610039] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 0.613018] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
[ 0.618102] clocksource: Switched to clocksource xen
[ 0.620401] pnp: PnP ACPI init
[ 0.622684] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[ 0.625049] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 0.625196] system 00:01: [io 0x08a0-0x08a3] has been reserved
[ 0.627392] system 00:01: [io 0x0cc0-0x0ccf] has been reserved
[ 0.629647] system 00:01: [io 0x04d0-0x04d1] has been reserved
[ 0.631868] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 0.631905] xen: --> pirq=17 -> irq=8 (gsi=8)
[ 0.631948] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 0.631977] xen: --> pirq=18 -> irq=12 (gsi=12)
[ 0.632051] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
[ 0.632080] xen: --> pirq=19 -> irq=1 (gsi=1)
[ 0.632123] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[ 0.632152] xen: --> pirq=20 -> irq=6 (gsi=6)
[ 0.632159] pnp 00:05: [dma 2]
[ 0.632195] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
[ 0.632228] xen: --> pirq=21 -> irq=4 (gsi=4)
[ 0.632270] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[ 0.632311] xen: --> pirq=22 -> irq=7 (gsi=7)
[ 0.632345] pnp 00:07: Plug and Play ACPI device, IDs PNP0400 (active)
[ 0.632410] system 00:08: [io 0x10c0-0x1141] has been reserved
[ 0.634602] system 00:08: [io 0xb044-0xb047] has been reserved
[ 0.636861] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 0.645085] pnp: PnP ACPI: found 9 devices
[ 0.656479] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 0.660376] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[ 0.660381] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[ 0.660386] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[ 0.660390] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
[ 0.660439] NET: Registered protocol family 2
[ 0.662726] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[ 0.666602] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[ 0.668924] TCP: Hash tables configured (established 16384 bind 16384)
[ 0.671140] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[ 0.673319] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[ 0.675576] NET: Registered protocol family 1
[ 0.677894] RPC: Registered named UNIX socket transport module.
[ 0.680052] RPC: Registered udp transport module.
[ 0.682163] RPC: Registered tcp transport module.
[ 0.684279] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.686477] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.688676] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.690882] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.693165] pci 0000:00:02.0: Video device with shadowed ROM
[ 0.693234] PCI: CLS 0 bytes, default 64
[ 0.693340] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2b3ee9f8ea7, max_idle_ns: 440795347344 ns
[ 0.698064] futex hash table entries: 2048 (order: 5, 131072 bytes)
[ 0.700405] audit: initializing netlink subsys (disabled)
[ 0.702627] audit: type=2000 audit(1445803843.255:1): initialized
[ 0.705217] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.711079] VFS: Disk quotas dquot_6.6.0
[ 0.713353] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.716422] NFS: Registering the id_resolver key type
[ 0.718566] Key type id_resolver registered
[ 0.720690] Key type id_legacy registered
[ 0.722912] SGI XFS with ACLs, security attributes, no debug enabled
[ 0.732714] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[ 0.736562] io scheduler noop registered
[ 0.738671] io scheduler deadline registered
[ 0.740814] io scheduler cfq registered (default)
[ 0.743074] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 0.745443] uvesafb: failed to execute /sbin/v86d
[ 0.747577] uvesafb: make sure that the v86d helper is installed and executable
[ 0.751422] uvesafb: Getting VBE info block failed (eax=0x4f00, err=-2)
[ 0.753615] uvesafb: vbe_init() failed with -22
[ 0.755710] uvesafb: probe of uvesafb.0 failed with error -22
[ 0.758641] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.764593] ACPI: Power Button [PWRF]
[ 0.767911] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
[ 0.773840] ACPI: Sleep Button [SLPF]
[ 0.777655] xen:xen_evtchn: Event-channel device installed
[ 0.781629] xen: --> pirq=23 -> irq=28 (gsi=28)
[ 0.781783] xen:grant_table: Grant tables using version 1 layout
[ 0.786943] Grant table initialized
[ 0.792598] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.828186] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 0.835759] Non-volatile memory driver v1.3
[ 0.839144] Linux agpgart interface v0.103
[ 0.842568] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[ 0.845865] AMD IOMMUv2 functionality not available on this system
[ 0.849200] [drm] Initialized drm 1.1.0 20060810
[ 0.853115] [drm] Found bochs VGA, ID 0xb0c4.
[ 0.856372] [drm] Framebuffer size 8192 kB @ 0xf1000000, ioports @ 0x1ce.
[ 0.859788] [TTM] Zone kernel: Available graphics memory: 1021590 kiB
[ 0.863120] [TTM] Initializing pool allocator
[ 0.866389] [TTM] Initializing DMA pool allocator
[ 6.816112] random: nonblocking pool is initialized
[ 21.846327] INFO: rcu_sched self-detected stall on CPU { 1} (t=23321 jiffies g=-238 c=-239 q=27)
[ 21.846327] rcu_sched kthread starved for 18649 jiffies! g18446744073709551378 c18446744073709551377 f0x0
[ 21.846327] Task dump for CPU 1:
[ 21.846327] swapper/0 R running task 0 1 0 0x00000008
[ 21.846327] ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001
[ 21.846327] ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083
[ 21.846327] 0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940
[ 21.846327] Call Trace:
[ 21.846327] <IRQ> [<ffffffff810756ba>] sched_show_task+0xaa/0x110
[ 21.846327] [<ffffffff810778a8>] dump_cpu_task+0x38/0x40
[ 21.846327] [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0
[ 21.846327] [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0
[ 21.846327] [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20
[ 21.846327] [<ffffffff8107832c>] ? account_system_time+0x7c/0x120
[ 21.846327] [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 21.846327] [<ffffffff8109a994>] update_process_times+0x34/0x60
[ 21.846327] [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40
[ 21.846327] [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80
[ 21.846327] [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130
[ 21.846327] [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0
[ 21.846327] [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140
[ 21.846327] [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100
[ 21.846327] [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50
[ 21.846327] [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40
[ 21.846327] [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150
[ 21.846327] [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80
[ 21.846327] [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50
[ 21.846327] [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70
[ 21.846327] <EOI> [<ffffffff812eb964>] ? __memset+0x24/0x30
[ 21.846327] [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650
[ 21.846327] [<ffffffff814028b3>] bochs_bo_move+0x13/0x20
[ 21.846327] [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630
[ 21.846327] [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340
[ 21.846327] [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0
[ 21.846327] [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0
[ 21.846327] [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160
[ 21.846327] [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0
[ 21.846327] [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0
[ 21.846327] [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0
[ 21.846327] [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0
[ 21.846327] [<ffffffff814027c7>] bochs_load+0x87/0x90
[ 21.846327] [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100
[ 21.846327] [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0
[ 21.846327] [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0
[ 21.846327] [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0
[ 21.846327] [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110
[ 21.846327] [<ffffffff8130e812>] pci_device_probe+0xc2/0x110
[ 21.846327] [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0
[ 21.846327] [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90
[ 21.846327] [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0
[ 21.846327] [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90
[ 21.846327] [<ffffffff8140bd09>] driver_attach+0x19/0x20
[ 21.846327] [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220
[ 21.846327] [<ffffffff8140cbdb>] driver_register+0x5b/0xe0
[ 21.846327] [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50
[ 21.846327] [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100
[ 21.846327] [<ffffffff81990424>] ? ttm_init+0x6a/0x6a
[ 21.846327] [<ffffffff8199043b>] bochs_init+0x17/0x19
[ 21.846327] [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0
[ 21.846327] [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec
[ 21.846327] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 21.846327] [<ffffffff816875b9>] kernel_init+0x9/0xe0
[ 21.846327] [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70
[ 21.846327] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 84.962072] INFO: rcu_sched self-detected stall on CPU { 1} (t=86423 jiffies g=-238 c=-239 q=27)
[ 84.962072] rcu_sched kthread starved for 81751 jiffies! g18446744073709551378 c18446744073709551377 f0x0
[ 84.962072] Task dump for CPU 1:
[ 84.962072] swapper/0 R running task 0 1 0 0x00000008
[ 84.962072] ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001
[ 84.962072] ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083
[ 84.962072] 0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940
[ 84.962072] Call Trace:
[ 84.962072] <IRQ> [<ffffffff810756ba>] sched_show_task+0xaa/0x110
[ 84.962072] [<ffffffff810778a8>] dump_cpu_task+0x38/0x40
[ 84.962072] [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0
[ 84.962072] [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0
[ 84.962072] [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20
[ 84.962072] [<ffffffff8107832c>] ? account_system_time+0x7c/0x120
[ 84.962072] [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 84.962072] [<ffffffff8109a994>] update_process_times+0x34/0x60
[ 84.962072] [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40
[ 84.962072] [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80
[ 84.962072] [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130
[ 84.962072] [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0
[ 84.962072] [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140
[ 84.962072] [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100
[ 84.962072] [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50
[ 84.962072] [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40
[ 84.962072] [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150
[ 84.962072] [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80
[ 84.962072] [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50
[ 84.962072] [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70
[ 84.962072] <EOI> [<ffffffff812eb964>] ? __memset+0x24/0x30
[ 84.962072] [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650
[ 84.962072] [<ffffffff814028b3>] bochs_bo_move+0x13/0x20
[ 84.962072] [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630
[ 84.962072] [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340
[ 84.962072] [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0
[ 84.962072] [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0
[ 84.962072] [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160
[ 84.962072] [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0
[ 84.962072] [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0
[ 84.962072] [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0
[ 84.962072] [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0
[ 84.962072] [<ffffffff814027c7>] bochs_load+0x87/0x90
[ 84.962072] [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100
[ 84.962072] [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0
[ 84.962072] [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0
[ 84.962072] [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0
[ 84.962072] [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110
[ 84.962072] [<ffffffff8130e812>] pci_device_probe+0xc2/0x110
[ 84.962072] [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0
[ 84.962072] [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90
[ 84.962072] [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0
[ 84.962072] [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90
[ 84.962072] [<ffffffff8140bd09>] driver_attach+0x19/0x20
[ 84.962072] [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220
[ 84.962072] [<ffffffff8140cbdb>] driver_register+0x5b/0xe0
[ 84.962072] [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50
[ 84.962072] [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100
[ 84.962072] [<ffffffff81990424>] ? ttm_init+0x6a/0x6a
[ 84.962072] [<ffffffff8199043b>] bochs_init+0x17/0x19
[ 84.962072] [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0
[ 84.962072] [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec
[ 84.962072] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 84.962072] [<ffffffff816875b9>] kernel_init+0x9/0xe0
[ 84.962072] [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70
[ 84.962072] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 150.003525] INFO: rcu_sched self-detected stall on CPU { 1} (t=151445 jiffies g=-238 c=-239 q=27)
[ 150.003525] rcu_sched kthread starved for 146773 jiffies! g18446744073709551378 c18446744073709551377 f0x0
[ 150.003525] Task dump for CPU 1:
[ 150.003525] swapper/0 R running task 0 1 0 0x00000008
[ 150.003525] ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001
[ 150.003525] ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083
[ 150.003525] 0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940
[ 150.003525] Call Trace:
[ 150.003525] <IRQ> [<ffffffff810756ba>] sched_show_task+0xaa/0x110
[ 150.003525] [<ffffffff810778a8>] dump_cpu_task+0x38/0x40
[ 150.003525] [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0
[ 150.003525] [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0
[ 150.003525] [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20
[ 150.003525] [<ffffffff8107832c>] ? account_system_time+0x7c/0x120
[ 150.003525] [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 150.003525] [<ffffffff8109a994>] update_process_times+0x34/0x60
[ 150.003525] [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40
[ 150.003525] [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80
[ 150.003525] [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130
[ 150.003525] [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0
[ 150.003525] [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140
[ 150.003525] [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100
[ 150.003525] [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50
[ 150.003525] [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40
[ 150.003525] [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150
[ 150.003525] [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80
[ 150.003525] [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50
[ 150.003525] [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70
[ 150.003525] <EOI> [<ffffffff812eb964>] ? __memset+0x24/0x30
[ 150.003525] [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650
[ 150.003525] [<ffffffff814028b3>] bochs_bo_move+0x13/0x20
[ 150.003525] [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630
[ 150.003525] [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340
[ 150.003525] [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0
[ 150.003525] [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0
[ 150.003525] [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160
[ 150.003525] [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0
[ 150.003525] [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0
[ 150.003525] [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0
[ 150.003525] [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0
[ 150.003525] [<ffffffff814027c7>] bochs_load+0x87/0x90
[ 150.003525] [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100
[ 150.003525] [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0
[ 150.003525] [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0
[ 150.003525] [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0
[ 150.003525] [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110
[ 150.003525] [<ffffffff8130e812>] pci_device_probe+0xc2/0x110
[ 150.003525] [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0
[ 150.003525] [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90
[ 150.003525] [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0
[ 150.003525] [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90
[ 150.003525] [<ffffffff8140bd09>] driver_attach+0x19/0x20
[ 150.003525] [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220
[ 150.003525] [<ffffffff8140cbdb>] driver_register+0x5b/0xe0
[ 150.003525] [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50
[ 150.003525] [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100
[ 150.003525] [<ffffffff81990424>] ? ttm_init+0x6a/0x6a
[ 150.003525] [<ffffffff8199043b>] bochs_init+0x17/0x19
[ 150.003525] [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0
[ 150.003525] [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec
[ 150.003525] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 150.003525] [<ffffffff816875b9>] kernel_init+0x9/0xe0
[ 150.003525] [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70
[ 150.003525] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 215.286173] INFO: rcu_sched self-detected stall on CPU { 1} (t=216742 jiffies g=-238 c=-239 q=27)
[ 215.286173] rcu_sched kthread starved for 212070 jiffies! g18446744073709551378 c18446744073709551377 f0x0
[ 215.286173] Task dump for CPU 1:
[ 215.286173] swapper/0 R running task 0 1 0 0x00000008
[ 215.286173] ffffffff818e4740 ffff88007f443c08 ffffffff810756ba 0000000000000001
[ 215.286173] ffffffff818e4740 ffff88007f443c28 ffffffff810778a8 0000000000000083
[ 215.286173] 0000000000000002 ffff88007f443c58 ffffffff81094df0 ffff88007f455940
[ 215.286173] Call Trace:
[ 215.286173] <IRQ> [<ffffffff810756ba>] sched_show_task+0xaa/0x110
[ 215.286173] [<ffffffff810778a8>] dump_cpu_task+0x38/0x40
[ 215.286173] [<ffffffff81094df0>] rcu_dump_cpu_stacks+0x80/0xc0
[ 215.286173] [<ffffffff81097cae>] rcu_check_callbacks+0x3ce/0x6a0
[ 215.286173] [<ffffffff810d3937>] ? acct_account_cputime+0x17/0x20
[ 215.286173] [<ffffffff8107832c>] ? account_system_time+0x7c/0x120
[ 215.286173] [<ffffffff81085f21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 215.286173] [<ffffffff8109a994>] update_process_times+0x34/0x60
[ 215.286173] [<ffffffff810a8551>] tick_sched_handle.isra.17+0x31/0x40
[ 215.286173] [<ffffffff810a859c>] tick_sched_timer+0x3c/0x80
[ 215.286173] [<ffffffff8109b00b>] __hrtimer_run_queues+0xab/0x130
[ 215.286173] [<ffffffff8109b5a3>] hrtimer_interrupt+0xa3/0x1a0
[ 215.286173] [<ffffffff8100879e>] xen_timer_interrupt+0x2e/0x140
[ 215.286173] [<ffffffff8108d67c>] handle_irq_event_percpu+0x5c/0x100
[ 215.286173] [<ffffffff810908e9>] handle_percpu_irq+0x39/0x50
[ 215.286173] [<ffffffff8108ce47>] generic_handle_irq+0x27/0x40
[ 215.286173] [<ffffffff8137b30b>] evtchn_fifo_handle_events+0x14b/0x150
[ 215.286173] [<ffffffff813783a2>] __xen_evtchn_do_upcall+0x42/0x80
[ 215.286173] [<ffffffff8137a02f>] xen_evtchn_do_upcall+0x2f/0x50
[ 215.286173] [<ffffffff81696ccb>] xen_hvm_callback_vector+0x6b/0x70
[ 215.286173] <EOI> [<ffffffff812eb964>] ? __memset+0x24/0x30
[ 215.286173] [<ffffffff813fd88a>] ? ttm_bo_move_memcpy+0x5da/0x650
[ 215.286173] [<ffffffff814028b3>] bochs_bo_move+0x13/0x20
[ 215.286173] [<ffffffff813fae63>] ttm_bo_handle_move_mem+0x2a3/0x630
[ 215.286173] [<ffffffff813fb903>] ? ttm_bo_mem_space+0x163/0x340
[ 215.286173] [<ffffffff813fbf82>] ttm_bo_validate+0x1c2/0x1d0
[ 215.286173] [<ffffffff81080001>] ? pull_rt_task+0x111/0x2f0
[ 215.286173] [<ffffffff81402e76>] bochs_bo_pin+0xe6/0x160
[ 215.286173] [<ffffffff81403c4d>] bochsfb_create+0xfd/0x3c0
[ 215.286173] [<ffffffff813cb659>] ? drm_helper_probe_single_connector_modes_merge_bits+0x179/0x4d0
[ 215.286173] [<ffffffff813d695e>] drm_fb_helper_initial_config+0x1de/0x3d0
[ 215.286173] [<ffffffff81403f86>] bochs_fbdev_init+0x76/0xa0
[ 215.286173] [<ffffffff814027c7>] bochs_load+0x87/0x90
[ 215.286173] [<ffffffff813df9aa>] drm_dev_register+0xaa/0x100
[ 215.286173] [<ffffffff813e200e>] drm_get_pci_dev+0xce/0x1e0
[ 215.286173] [<ffffffff8140285e>] bochs_pci_probe+0x8e/0xa0
[ 215.286173] [<ffffffff8130e6f0>] local_pci_probe+0x40/0xa0
[ 215.286173] [<ffffffff8130e65f>] ? pci_match_device+0xef/0x110
[ 215.286173] [<ffffffff8130e812>] pci_device_probe+0xc2/0x110
[ 215.286173] [<ffffffff8140c181>] driver_probe_device+0x131/0x2d0
[ 215.286173] [<ffffffff8140c3ab>] __driver_attach+0x8b/0x90
[ 215.286173] [<ffffffff8140c320>] ? driver_probe_device+0x2d0/0x2d0
[ 215.286173] [<ffffffff8140a3ff>] bus_for_each_dev+0x5f/0x90
[ 215.286173] [<ffffffff8140bd09>] driver_attach+0x19/0x20
[ 215.286173] [<ffffffff8140b95b>] bus_add_driver+0x1ab/0x220
[ 215.286173] [<ffffffff8140cbdb>] driver_register+0x5b/0xe0
[ 215.286173] [<ffffffff8130dea7>] __pci_register_driver+0x47/0x50
[ 215.286173] [<ffffffff813e21f6>] drm_pci_init+0xd6/0x100
[ 215.286173] [<ffffffff81990424>] ? ttm_init+0x6a/0x6a
[ 215.286173] [<ffffffff8199043b>] bochs_init+0x17/0x19
[ 215.286173] [<ffffffff810020f3>] do_one_initcall+0x83/0x1b0
[ 215.286173] [<ffffffff8195e04e>] kernel_init_freeable+0x164/0x1ec
[ 215.286173] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 215.286173] [<ffffffff816875b9>] kernel_init+0x9/0xe0
[ 215.286173] [<ffffffff81695a5f>] ret_from_fork+0x3f/0x70
[ 215.286173] [<ffffffff816875b0>] ? rest_init+0x80/0x80
[ 225.423476] fbcon: bochsdrmfb (fb0) is primary device
[ 225.511319] Console: switching to colour frame buffer device 128x48
[ 225.513829] bochs-drm 0000:00:02.0: fb0: bochsdrmfb frame buffer device
[ 225.513867] bochs-drm 0000:00:02.0: registered panic notifier
[ 225.517061] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:02.0 on minor 0
[ 225.519905] loop: module loaded
[ 225.521733] ata_piix 0000:00:01.1: version 2.13
[ 225.523390] scsi host0: ata_piix
[ 225.523836] scsi host1: ata_piix
[ 225.523937] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc200 irq 14
[ 225.523978] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc208 irq 15
[ 225.524238] xen_netfront: Initialising Xen virtual ethernet driver
[ 225.526210] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 225.526258] ehci-pci: EHCI PCI platform driver
[ 225.526302] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 225.526338] ohci-pci: OHCI PCI platform driver
[ 225.526377] uhci_hcd: USB Universal Host Controller Interface driver
[ 225.526464] usbcore: registered new interface driver usblp
[ 225.526516] usbcore: registered new interface driver usb-storage
[ 225.526600] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[ 225.530314] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 225.530453] blkfront: xvda: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled;
[ 225.532200] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 225.533481] mousedev: PS/2 mouse device common for all mice
[ 225.537090] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[ 225.537147] input: Xen Virtual Keyboard as /devices/virtual/input/input3
[ 225.537236] xvda: xvda1
[ 225.540348] input: Xen Virtual Pointer as /devices/virtual/input/input4
[ 225.549960] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[ 225.551740] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
[ 225.553557] xen_wdt: Xen WatchDog Timer Driver v0.01
[ 225.555468] xen_wdt: initialized (timeout=60s, nowayout=1)
[ 225.556634] watchdog: Software Watchdog: cannot register miscdev on minor=130 (err=-16).
[ 225.557701] watchdog: Software Watchdog: a legacy watchdog module is probably present.
[ 225.559965] softdog: Software Watchdog Timer: 0.08 initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=1)
[ 225.562587] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com
[ 225.564468] hidraw: raw HID events driver (C) Jiri Kosina
[ 225.566469] usbcore: registered new interface driver usbhid
[ 225.567780] usbhid: USB HID core driver
[ 225.570207] NET: Registered protocol family 26
[ 225.571531] Netfilter messages via NETLINK v0.30.
[ 225.572836] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[ 225.574602] ctnetlink v0.93: registering with nfnetlink.
[ 225.576176] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 225.577482] geneve_core: Geneve core logic
[ 225.578632] Initializing XFRM netlink socket
[ 225.579773] NET: Registered protocol family 17
[ 225.580967] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[ 225.583316] l2tp_core: L2TP core driver, V2.0
[ 225.584519] Key type dns_resolver registered
[ 225.586190] registered taskstats version 1
[ 225.588223] Magic number: 3:642:195
[ 225.589786] acpi device:50: hash matches
[ 225.591260] ALSA device list:
[ 225.592563] No soundcards found.
[ 226.175106] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input6
[ 226.177509] md: Waiting for all devices to be available before autodetect
[ 226.178885] md: If you don't use raid, use raid=noautodetect
[ 226.180794] md: Autodetecting RAID arrays.
[ 226.182167] md: Scanned 0 and added 0 devices.
[ 226.183556] md: autorun ...
[ 226.184866] md: ... autorun DONE.
[ 226.188611] XFS (xvda1): Mounting V4 Filesystem
[ 226.758530] XFS (xvda1): Ending clean mount
[ 226.759888] VFS: Mounted root (xfs filesystem) readonly on device 202:1.
[ 226.761820] devtmpfs: mounted
[ 226.764661] Freeing unused kernel memory: 1000K (ffffffff81947000 - ffffffff81a41000)
[ 228.118811] systemd-udevd[1442]: starting version 216
[ 228.254569] FDC 0 is a S82078B
|
This error delayed kernel boot about 3 minute but after booting system worked
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
TigerJr
Guru
Joined: 19 Jun 2007
Posts: 540
|
| Posted: Sun Nov 08, 2015 2:06 pm Post subject: |
|
|
Found more INFO: rcu_sched self-detected stall on CPU :
https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
| Quote: | What Causes RCU CPU Stall Warnings?
So your kernel printed an RCU CPU stall warning. The next question is
"What caused it?" The following problems can result in RCU CPU stall warnings:
o A CPU looping in an RCU read-side critical section.
o A CPU looping with interrupts disabled. This condition can result in RCU-sched and RCU-bh stalls.
o A CPU looping with preemption disabled. This condition can result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh stalls.
o A CPU looping with bottom halves disabled. This condition can result in RCU-sched and RCU-bh stalls.
o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel without invoking schedule(). Note that cond_resched() does not necessarily prevent RCU CPU stall warnings. Therefore, if the looping in the kernel is really expected and desirable behavior, you might need to replace some of the cond_resched() calls with calls to cond_resched_rcu_qs().
o Anything that prevents RCU's grace-period kthreads from running. This can result in the "All QSes seen" console-log message. This message will include information on when the kthread last ran and how often it should be expected to run.
o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might happen to preempt a low-priority task in the middle of an RCU read-side critical section. This is especially damaging if that low-priority task is not permitted to run on any other CPU, in which case the next RCU grace period can never complete, which will eventually cause the system to run out of memory and hang. While the system is in the process of running itself out of memory, you might see stall-warning messages.
o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that is running at a higher priority than the RCU softirq threads. This will prevent RCU callbacks from ever being invoked, and in a CONFIG_PREEMPT_RCU kernel will further prevent RCU grace periods from ever completing. Either way, the system will eventually run out of memory and hang. In the CONFIG_PREEMPT_RCU case, you might see stall-warning messages.
o A hardware or software issue shuts off the scheduler-clock interrupt on a CPU that is not in dyntick-idle mode. This problem really has happened, and seems to be most likely to result in RCU CPU stall warnings for CONFIG_NO_HZ_COMMON=n kernels.
o A bug in the RCU implementation.
o A hardware failure. This is quite unlikely, but has occurred at least once in real life. A CPU failed in a running system, becoming unresponsive, but not causing an immediate crash. This resulted in a series of RCU CPU stall warnings, eventually leading the realization that the CPU had failed. |
_________________
Do not use gentoo, it die |
|
| Back to top |
|
|
reith
n00b
Joined: 13 Nov 2015
Posts: 16
|
| Posted: Fri Nov 13, 2015 7:49 pm Post subject: |
|
|
| Very similar crash dump here, But I have a core i3 haswell CPU. Currently running on 4.0.5 kernel. I booted in EFI mode and set SMP on. |
|
| Back to top |
|
|
Polyatomic
n00b
Joined: 18 May 2014
Posts: 36
|
| Posted: Sun Nov 15, 2015 9:07 pm Post subject: |
|
|
reith, May I?
Could you please paste site service some of the system log. Is your rig on the magnanimous
init factotem?. Say `journalctl -b -1`(-0 is the last boot, -1 the boot before last) You can get a
list with `journalctl --list-boots` . Of course you prolly new that already, I'm thinking that maybe
a careful observer can analyse the crapout. Prolly not use pastebin.com, its veritable crud. ) |
|
| Back to top |
|
|
reith
n00b
Joined: 13 Nov 2015
Posts: 16
|
| Posted: Mon Nov 16, 2015 5:32 pm Post subject: |
|
|
| Polyatomic wrote: |
Could you please paste site service some of the system log. Is your rig on the magnanimous
init factotem?. Say `journalctl -b -1`(-0 is the last boot, -1 the boot before last) You can get a
list with `journalctl --list-boots` . Of course you prolly new that already, I'm thinking that maybe
a careful observer can analyse the crapout. Prolly not use pastebin.com, its veritable crud. ) |
I'm not using systemd. Problem happens after system is running for more than 10 hours; so I may see the problem in weekend.
Also I blacklisted nouveau and It's possible I don't see problem again, hopefully.
This is log for a previous crash, but there is no report about possible malfunctionings before it[/code]:
| Code: |
Nov 13 19:34:26 localhost kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 3, t=60002 jiffies, g=194181, c=194180, q=4450)
Nov 13 19:34:26 localhost kernel: Task dump for CPU 2:
Nov 13 19:34:26 localhost kernel: swapper/2 R running task 31424 0 1 0x00000008
Nov 13 19:34:26 localhost kernel: 0000000000000010 0000000000000246 ffff88010ac47e68 0000000000000018
Nov 13 19:34:26 localhost kernel: ffffffff815ec4c1 000009d91b6d951f ffffffff81cb96f8 ffff88010ac40000
Nov 13 19:34:26 localhost kernel: ffffffff81d15140 ffff88010bd24300 ffffffff81cb9500 ffff88010ac40000
Nov 13 19:34:26 localhost kernel: Call Trace:
Nov 13 19:34:26 localhost kernel: [<ffffffff815ec4c1>] ? cpuidle_enter_state+0x61/0x160
Nov 13 19:34:26 localhost kernel: [<ffffffff815ec5e2>] ? cpuidle_enter+0x12/0x20
Nov 13 19:34:26 localhost kernel: [<ffffffff81086b78>] ? cpu_startup_entry+0x2b8/0x3c0
Nov 13 19:34:26 localhost kernel: [<ffffffff81030318>] ? start_secondary+0x138/0x140
Nov 13 19:34:29 localhost kernel: sysrq: SysRq : Keyboard mode set to system default
Nov 13 19:34:45 localhost syslog-ng[2309]: syslog-ng shutting down; version='3.7.2'
Nov 13 19:34:45 localhost dhcpcd[3017]: received SIGTERM, stopping
Nov 13 19:34:45 localhost dhcpcd[3017]: eno1: removing interface
Nov 13 19:34:45 localhost dictd[2567]: :I: SIGTERM: c/f = 47/0; 03:01:29r 0.000u 0.000s
Nov 13 19:34:30 localhost kernel: sysrq: SysRq : Terminate All Tasks
Nov 13 19:34:45 localhost vnstatd[2708]: SIGTERM received, exiting.
|
|
|
| Back to top |
|
|
fredbear5150
Tux's lil' helper
Joined: 11 Oct 2003
Posts: 113
|
| Posted: Thu Nov 26, 2015 3:15 pm Post subject: |
|
|
I am seeing a similar issue on a new build of an AMD Sempron Triple Core CPU. The PC was previously running Gentoo with no problems, it was in a cupboard for a few months and was then dragged out and rebuilt from new. Ever since that rebuild, it has crashed after 2-3 hours use.
It does seem to be during times of "reasonably" heavy loads, it certainly went down a couple of times when I was running Chrome, for the last couple of days I have been rsync-ing my NAS drive to it preparing that for a rebuild.
Here's what's in my /var/log/messages:
Nov 26 14:53:27 arcturus kernel: Call Trace:
Nov 26 14:53:27 arcturus kernel: [<ffffffff8100cc2b>] ? default_idle+0x1b/0xc0
Nov 26 14:53:27 arcturus kernel: [<ffffffff8100d6fa>] ? arch_cpu_idle+0xa/0x10
Nov 26 14:53:27 arcturus kernel: [<ffffffff8108b0c6>] ? cpu_startup_entry+0x2c6/0x390
Nov 26 14:53:27 arcturus kernel: [<ffffffff810346f3>] ? start_secondary+0x123/0x130
Nov 26 14:56:27 arcturus kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=780022 jiffies, g=44399, c=44398, q=6563)
Nov 26 14:56:27 arcturus kernel: Task dump for CPU 2:
Nov 26 14:56:27 arcturus kernel: swapper/2 R running task 15440 0 1 0x00000008
Nov 26 14:56:27 arcturus kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Nov 26 14:56:27 arcturus kernel: 0000000000000000 0000000000000000 ffffffffffffff2e ffffffff8100cc2b
Nov 26 14:56:27 arcturus kernel: 0000000000000010 0000000000000246 ffff88011ab67ea8 0000000000000018
Sometimes the system locks completely, other times the screen freezes though the mouse pointer still moves, you cannot click anything though.
It also appears to always be on the "last" CPU in my three cores, that maybe a common factor with it being on CPU 3 in the Quad Core CPU.
I'm running kernel 4.1.12, here's my emerge --info:
Portage 2.2.20.1 (python 3.4.3-final-0, default/linux/amd64/13.0/desktop, gcc-4.9.3, glibc-2.21-r1, 4.1.12-gentoo x86_64)
=================================================================
System uname: Linux-4.1.12-gentoo-x86_64-AMD_Phenom-tm-_8400_Triple-Core_Processor-with-gentoo-2.2
KiB Mem: 4049008 total, 2274708 free
KiB Swap: 2621432 total, 2621432 free
Timestamp of repository gentoo: Thu, 26 Nov 2015 09:15:01 +0000
sh bash 4.3_p39
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash: 4.3_p39::gentoo
dev-java/java-config: 2.2.0::gentoo
dev-lang/perl: 5.20.2::gentoo
dev-lang/python: 2.7.10-r1::gentoo, 3.4.3::gentoo
dev-util/cmake: 3.3.1-r1::gentoo
dev-util/pkgconfig: 0.28-r2::gentoo
sys-apps/baselayout: 2.2::gentoo
sys-apps/openrc: 0.17::gentoo
sys-apps/sandbox: 2.6-r1::gentoo
sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo
sys-devel/automake: 1.10.3-r1::gentoo, 1.11.6-r1::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils: 2.25.1-r1::gentoo
sys-devel/gcc: 4.9.3::gentoo
sys-devel/gcc-config: 1.7.3::gentoo
sys-devel/libtool: 2.4.6::gentoo
sys-devel/make: 4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc: 2.21-r1::gentoo
Repositories:
gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.uk.gentoo.org/gentoo-portage
priority: -1000
steam-overlay
location: /var/lib/layman/steam-overlay
masters: gentoo
priority: 0
sunrise
location: /var/lib/layman/sunrise
masters: gentoo
priority: 1
rion
location: /var/lib/layman/rion
masters: gentoo
priority: 2
bumblebee
location: /var/lib/layman/bumblebee
masters: gentoo
priority: 3
games-overlay
location: /var/lib/layman/games-overlay
masters: gentoo
priority: 4
init6
location: /var/lib/layman/init6
masters: gentoo
priority: 5
mate
location: /var/lib/layman/mate
masters: gentoo
priority: 6
jorgicio
location: /var/lib/layman/jorgicio
masters: gentoo
priority: 7
sakaki-tools
location: /usr/local/portage/sakaki-tools
sync-type: git
sync-uri: https://github.com/sakaki-/sakaki-tools.git
masters: gentoo
priority: 50
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=amdfam10 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=amdfam10 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=3"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://mirror.bytemark.co.uk/gentoo/ http://mirror.bytemark.co.uk/gentoo/ rsync://mirror.bytemark.co.uk/gentoo/ http://mirror.qubenet.net/mirror/gentoo/ ftp://ftp.mirrorservice.org/sites/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4 -l3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow 3dnowext X a52 aac acl acpi alsa amd64 avahi berkdb bindist bluetooth branding bzip2 cairo cdda cddb cdparanoia cdr cli consolekit cracklib crypt css cups cxx dbus djvu dri dts dvd dvdr emboss encode exif fam ffmpeg firefox flac fortran gdbm gif glamor gnome-keyring gphoto2 gpm gtk iconv icu ipv6 java javascript jpeg jpeg2k lame lcms ldap libnotify lm_sensors lock mad mmx mmxext mng modules mp3 mp4 mpeg mtp multilib musepack ncurses nls nptl nsplugin ogg opengl openmp pam pango pcre pdf perl php png policykit popcnt ppds pulseaudio python qt3support readline ruby samba sdl seccomp session speex spell sse sse2 sse3 sse4a ssl startup-notification svg tagwriting tcpd tiff timidity truetype udev udisks unicode upnp upnp-av upower usb vorbis wavpack webkit wmf wxwidgets x264 xattr xcb xml xv xvid zeroconf zlib" ABI_X86="32 64" ALSA_CARDS="cs46xx hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" SANE_BACKENDS="plustek canon hp" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
