Qemu/KVM unstable

[grooveman]

Hello.

I've had this problem now for over 2 months. I'm not sure if it is an issue with the kernel, qemu, or windows, but I'm having some pretty serious stability issues with my qemu win10 machine.

Because I wear a lot of hats at work, I have to have a windows system. I've been successfully using qemu/kvm for this for years. About 2 months ago I did an update, and found that my windows vm has been crashing regularly ever since. But, every day since then, without fail, my windows system has crashed multiple times.

Typically, it locks up when idle, though twice it just froze in the middle of writing emails or viewing an excel document. Sometimes it locks up 2 minutes after going idle, and other times is locks up 6 hours after going idle -- there doesn't seem to be any rhyme or reason to it. Furthermore, there doesn't seem to be any really useful information in the windows Event viewer, the qemu console, or on the Gentoo host system. I do get some chatter in /var/log/messages that might be related:

Code: Select all

Oct 25 16:46:00 My-PC kernel: RIP: 0033:0x7f7d80f51f37
Oct 25 16:46:00 My-PC kernel: RSP: 002b:00007f7d7a3d8998 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Oct 25 16:46:00 My-PC kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f7d80f51f37
Oct 25 16:46:00 My-PC kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
Oct 25 16:46:00 My-PC kernel: RBP: 0000000000000000 R08: 00005637e312e3a0 R09: 00007fff6739b080
Oct 25 16:46:00 My-PC kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00005637e324ca80
Oct 25 16:46:00 My-PC kernel: R13: 00007f7d84675000 R14: 0000000000000000 R15: 00005637e324ca80
Oct 25 16:46:00 My-PC kernel: qemu-system-x86: page allocation failure: order:4, mode:0x14080c0(GFP_KERNEL|__GFP_ZERO), nodemask=(null)
Oct 25 16:46:00 My-PC kernel: qemu-system-x86 cpuset=/ mems_allowed=0
Oct 25 16:46:00 My-PC kernel: CPU: 10 PID: 14794 Comm: qemu-system-x86 Tainted: P        W  O    4.14.65-gentoo #2
Oct 25 16:46:00 My-PC kernel: Hardware name: Dell Inc. Precision WorkStation T3500  /09KPNV, BIOS A13 10/30/2011
Oct 25 16:46:00 My-PC kernel: Call Trace:
Oct 25 16:46:00 My-PC kernel:  dump_stack+0x46/0x59
Oct 25 16:46:00 My-PC kernel:  warn_alloc+0xdb/0x170
Oct 25 16:46:00 My-PC kernel:  __alloc_pages_slowpath+0xd87/0xdf0
Oct 25 16:46:00 My-PC kernel:  ? native_set_ldt.part.8+0xa0/0xa0
Oct 25 16:46:00 My-PC kernel:  ? native_set_ldt.part.8+0xa0/0xa0
Oct 25 16:46:00 My-PC kernel:  __alloc_pages_nodemask+0x1ab/0x260
Oct 25 16:46:00 My-PC kernel:  dsalloc_pages+0x3a/0x70
Oct 25 16:46:00 My-PC kernel:  reserve_ds_buffers+0x10d/0x400
Oct 25 16:46:00 My-PC kernel:  x86_reserve_hardware+0x177/0x190
Oct 25 16:46:00 My-PC kernel:  x86_pmu_event_init+0x3d/0x220
Oct 25 16:46:00 My-PC kernel:  perf_try_init_event+0x31/0xa0
Oct 25 16:46:00 My-PC kernel:  perf_event_alloc+0x61a/0x7e0
Oct 25 16:46:00 My-PC kernel:  ? kvm_perf_overflow+0x40/0x40
Oct 25 16:46:00 My-PC kernel:  perf_event_create_kernel_counter+0x1f/0x140
Oct 25 16:46:00 My-PC kernel:  pmc_reprogram_counter+0xd6/0x120
Oct 25 16:46:00 My-PC kernel:  reprogram_gp_counter+0x11f/0x160
Oct 25 16:46:00 My-PC kernel:  kvm_pmu_handle_event+0x63/0xa0
Oct 25 16:46:00 My-PC kernel:  kvm_arch_vcpu_ioctl_run+0x109a/0x1610
Oct 25 16:46:00 My-PC kernel:  ? kvm_arch_vcpu_load+0x70/0x250
Oct 25 16:46:00 My-PC kernel:  ? handle_rdmsr+0x130/0x130
Oct 25 16:46:00 My-PC kernel:  ? kvm_arch_vcpu_load+0x8b/0x250
Oct 25 16:46:00 My-PC kernel:  ? kvm_vcpu_ioctl+0x2a6/0x570
Oct 25 16:46:00 My-PC kernel:  kvm_vcpu_ioctl+0x2a6/0x570
Oct 25 16:46:00 My-PC kernel:  ? do_futex+0x439/0xa00
Oct 25 16:46:00 My-PC kernel:  ? selinux_file_ioctl+0x84/0x1a0
Oct 25 16:46:00 My-PC kernel:  do_vfs_ioctl+0x8b/0x5e0
Oct 25 16:46:00 My-PC kernel:  ? security_file_ioctl+0x3f/0x60
Oct 25 16:46:00 My-PC kernel:  ? selinux_bprm_set_creds+0x290/0x290
Oct 25 16:46:00 My-PC kernel:  SyS_ioctl+0x6f/0x80
Oct 25 16:46:00 My-PC kernel:  do_syscall_64+0x61/0x110
Oct 25 16:46:00 My-PC kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Oct 25 16:46:00 My-PC kernel: RIP: 0033:0x7f7d80f51f37
Oct 25 16:46:00 My-PC kernel: RSP: 002b:00007f7d7a3d8998 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Oct 25 16:46:00 My-PC kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f7d80f51f37
Oct 25 16:46:00 My-PC kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
Oct 25 16:46:00 My-PC kernel: RBP: 0000000000000000 R08: 00005637e312e3a0 R09: 00007fff6739b080
Oct 25 16:46:00 My-PC kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00005637e324ca80
Oct 25 16:46:00 My-PC kernel: R13: 00007f7d84675000 R14: 0000000000000000 R15: 00005637e324ca80

I did memtest on my system... no errors cropped up (though I could try a longer scan... it is difficult to be without my system for hours on end). I also checked the hard drive, and didn't really see any issues there either (I point qemu to the entire HD instead of using a file image). I've tried at least 3 different kernels, and 3 different versions of qemu (currently on 3.0). The problem still persists. I've re-emerged my world numerous times since the problem manifested, hoping that maybe something just got a little out of sync. Nope. I've also dropped my KDE desktop temporarily, to make sure that somehow that was the problem (using i3 at the moment -- very minimal). Nope. All my qxl/virt drivers are up to date on the guest system.

I notice that the messages chatter says something about my memory being tainted in regard to starting up qemu... but I don't know what that means exactly... Some googling regarding the " qemu-system-x86: page allocation failure: order:4," portion found some bugs (fedora and debian), but they were fixed a while back... But I figure if this were the problem, I wouldn't be the only one experiencing it... and I've updated numerous times.

The odd thing is, that qemu itself does not die. It is the windows vm that locks up, and will not respond to anything other than a system_reset or system_powerdown. I can move around and issue commands within the qemu console as normal... but the VM is unresponsive (locked up).

Lastly, I do not use libvirt. I run qemu the old fashioned way... (and that's the way I likes it!). I'll include my startup command below.

Thanks.

My system:

emerge --info

Code: Select all

Portage 2.3.49 (python 3.6.5-final-0, default/linux/amd64/17.0/desktop/plasma, gcc-7.3.0, glibc-2.27-r6, 4.14.65-gentoo x86_64)
=================================================================
System uname: Linux-4.14.65-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_W3670_@_3.20GHz-with-gentoo-2.4.1
KiB Mem:    24677560 total,    304760 free
KiB Swap:   22906876 total,  22906876 free
Timestamp of repository gentoo: Wed, 31 Oct 2018 10:00:01 +0000
Head commit of repository gentoo: 098c792038f0411ce4e62a0bff02dae891b41348
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.38.3::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r4::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* @EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -mtune=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -mtune=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://lug.mtu.edu/gentoo/"
LANG="en_US"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en_US en"
MAKEOPTS="-j 7"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi activities alsa amd64 berkdb bluetooth bluray branding bzip2 cairo cdda cdr cleartype cli consolekit corefonts crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam ffmpeg flac fmpeg fortran frei0r fuse gdbm gif glamor gpm gtk iconv java jpeg kde kipi kvm kwallet lcms ldap libnotify libtirpc mad mmx mng mp3 mp4 mpeg ncurses nls nptl nvidia ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulse pulseaudio qml qt3support qt5 quicktime readline samba sdl seccomp semantic-desktop sftp spell sse sse2 ssl startup-notification svg tcpd threads tiff truetype type1 udev udf udisks unicode upower usb vdpau vorbis vpx widgets wxwidgets x264 x265 xattr xcb xcomposite xine xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="nvidia vga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

my qemu startup command:

Code: Select all

BUS=$(lsusb |grep Rainbow | awk '{print $2}' |awk '{print $1 + 0}')
BUSSY=$(echo $BUS + 0 |bc)
DEVY=$(lsusb |grep Rainbow | awk '{print $4}')
DEVV=$(echo ${DEVY::-1} |awk '{print $1 + 0}')
DEVVY=$(echo $DEVV + 0 |bc)
qemu-system-x86_64 -enable-kvm \
-cpu host \
-monitor stdio \
-name My-PC-WIN \
-vga qxl \
-drive file=/dev/sdc1,if=virtio,format=raw \
-net nic,model=virtio \
-net tap,ifname=tap0,script=no,downscript=no \
-device usb-tablet \
-m 16G \
-global qxl-vga.vram_size_mb=128 \
-global qxl-vga.ram_size_mb=128 \
-global qxl-vga.vgamem_mb=128 \
-smp cpus=4,cores=4,threads=1,sockets=1 \
-soundhw hda \
-usb \
-device usb-host,hostbus="$BUSSY",hostaddr="$DEVVY" \
-device virtio-serial-pci \
-spice port=5930,disable-ticketing \
-device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \
-chardev spicevmc,id=spicechannel0,name=vdagent \
-device virtio-serial \
-chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
-device qxl "$@"

current build of Qemu (though I have tried several):

Code: Select all

[ebuild   R   ~] app-emulation/qemu-3.0.0::gentoo 

my cpu:

Code: Select all

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2151.912
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2819.948
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 1
cpu cores       : 6
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 1904.480
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 2
cpu cores       : 6
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2848.781
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 8
cpu cores       : 6
apicid          : 16
initial apicid  : 16
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2003.351
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 9
cpu cores       : 6
apicid          : 18
initial apicid  : 18
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 3126.306
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 10
cpu cores       : 6
apicid          : 20
initial apicid  : 20
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2232.614
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2597.052
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 1
cpu cores       : 6
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 8
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2884.206
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 2
cpu cores       : 6
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 9
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 3130.473
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 8
cpu cores       : 6
apicid          : 17
initial apicid  : 17
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 10
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 2917.487
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 9
cpu cores       : 6
apicid          : 19
initial apicid  : 19
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 11
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           W3670  @ 3.20GHz
stepping        : 2
microcode       : 0x10
cpu MHz         : 3124.114
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 10
cpu cores       : 6
apicid          : 21
initial apicid  : 21
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6399.81
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bit[/quote]

Memory:
[quote]MemTotal:       24677560 kB
MemFree:          288912 kB
MemAvailable:    4398864 kB
Buffers:         4025280 kB
Cached:           386644 kB
SwapCached:            0 kB
Active:         19054664 kB
Inactive:        4788244 kB
Active(anon):   18084500 kB
Inactive(anon):  1412808 kB
Active(file):     970164 kB
Inactive(file):  3375436 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      22906876 kB
SwapFree:       22906876 kB
Dirty:             16240 kB
Writeback:             0 kB
AnonPages:      19279984 kB
Mapped:           292360 kB
Shmem:             66312 kB
Slab:             195468 kB
SReclaimable:     143484 kB
SUnreclaim:        51984 kB
KernelStack:       10368 kB
PageTables:        65712 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    35245656 kB
Committed_AS:   23428116 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:  17571840 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     3270628 kB
DirectMap2M:    19795968 kB
DirectMap1G:     2097152 kB

[LIsLinuxIsSogood]

grooveman, hey it's been a while since my trying to use qemu to boot from a physical disk, is that what you are doing here? anyway it seems due to your explanation of what's been tried that you pretty much can rule out the hard drive and kernel for now. (I would suggest something, but not sure if it is going to make a bit of difference)...In theory at least the command for starting qemu seems very specific and probably involves a lot of settings that the guest windows host won't agree with. At least my experience with virtual machines is that over time they do seem to break/die/fall apart a bit faster than their physical machine counterparts. Could you try modifying your launch command to see if other QEMU START COMMANDS WOULD HAVE ANY IMPACT ON THIS SITUATION??!

I don't actually know what command to pass, but it looks like you must have spent some time figuring out the exact start command for your system, or else you found it somewhere, and then modified that based on your custom needs for the command.

Could it be the mem or cpu settings on you qemu start command, apologies if this isn't as helpful, but wow look at that call trace being 20 lines!!

EDIT: Have you also attempted to send the log from the entire virtual machine startup process "captured" into a file somewhere? Then that could actually be informative as to whether or not the fix may involve something more permanent like a need to be doing a reinstallation of the guest...and hopefully not.)

[grooveman]

grooveman, hey it's been a while since my trying to use qemu to boot from a physical disk, is that what you are doing here? anyway it seems due to your explanation of what's been tried that you pretty much can rule out the hard drive and kernel for now. (I would suggest something, but not sure if it is going to make a bit of difference)...In theory at least the command for starting qemu seems very specific and probably involves a lot of settings that the guest windows host won't agree with. At least my experience with virtual machines is that over time they do seem to break/die/fall apart a bit faster than their physical machine counterparts. Could you try modifying your launch command to see if other QEMU START COMMANDS WOULD HAVE ANY IMPACT ON THIS SITUATION??!

I don't actually know what command to pass, but it looks like you must have spent some time figuring out the exact start command for your system, or else you found it somewhere, and then modified that based on your custom needs for the command.

Could it be the mem or cpu settings on you qemu start command, apologies if this isn't as helpful, but wow look at that call trace being 20 lines!!

EDIT: Have you also attempted to send the log from the entire virtual machine startup process "captured" into a file somewhere? Then that could actually be informative as to whether or not the fix may involve something more permanent like a need to be doing a reinstallation of the guest...and hopefully not.)

Hi LLISG,

The command evolved over time, but has been tried and true. I haven't made any changes in at least 6 months, maybe year.

I absolutely have tried to catch any output ... but there never is any. The windows system just suddenly freezes up -- no output anywhere. Nothing recorded in the event log, nothing on the qemu console... It makes it very difficult to trouble-shoot it. Again, it is the windows instance that has the problem. Qemu remains up and running.

Last night I ran a memtest for 15 hours... no errors... so I think I can rule that out. I try tweaking settings, but since there is no output, it is a shot in the dark. Since I'm at work, it is difficult to actually do my work, and trouble-shoot the windows instance (because I need it to do a large part of my work).

Since nothing changed with my system, or startup, something must have changed under the hood... I just cannot think of what it might be, and I cannot find any precedent for this behavior...

I guess, I will just have to try reducing my config... this could take a while... but I'll start by removing the video memory components...

[grooveman]

Ok.. I've ruled out bad RAM as an issue...

And I've reduced my qemu command to the barest of essentials:

Code: Select all

qemu-system-x86_64 -enable-kvm \
-cpu host \
-spice port=5930,disable-ticketing \
-monitor stdio \
-drive file=/dev/sdc1,if=virtio,format=raw \
-m 16G \

It is still crashing. I don't get it...

[Anon-E-moose]

Have you tried adding "-D <logfile>" to see if it will leave a hint?

ETA: man qemu and look for logfile to get several different options.

[grooveman]

I can certainly do that... but my understanding is that just takes what would have otherwise been sent to standard error... which I have been monitoring.

Can't hurt though... I"ll add it to the command. I appreciate the input.

[Anon-E-moose]

There are some debug options available depending on what you think might be causing the problem.
That's why I mentioned the man page.

ETA: Also https://fedoraproject.org/wiki/How_to_d ... n_problems, even though it's RH, it might give a clue or two.

[grooveman]

There are some debug options available depending on what you think might be causing the problem.
That's why I mentioned the man page.

ETA: Also https://fedoraproject.org/wiki/How_to_d ... n_problems, even though it's RH, it might give a clue or two.

Hi. Well, I did try logging, and as I thought, there is no output. The log file is zero bytes (Crashed twice).

I didn't really see much at that link... as they are using libvirt. I'm not. I'm also using windows for the guest... so none of the suggestions there really apply.

I have next to nothing to go on here... and this worked for YEARS. I just don't get it. The Qemu console reports nothing, and it is still operational. It is the windows guest OS that locks up. No errors...

[Anon-E-moose]

which version of qemu? Have you updated it lately, at least since the last time it worked?

[grooveman]

The problem has occurred ever since an update about 2.5 months ago now. I've tried every version of Qemu in portage during this time, and have updated numerous times between then and now (as well as every kernel update)... hoping that that some bug or incompatibility that might be responsible for this would be rectified... But this has not happened. I'm currently running Qemu 3.0, but it doesn't really seem to matter... I've stripped my command to the bare essentials...and it still does it.

I don't see how it could be something I'm doing... I was really hoping that someone was a aware of a new virtual driver for win 10, or a kernel patch, or new version of Qmeu or a new switch to qmeu that would resolve it...

I'll keep banging away at it...

Thanks.

[Anon-E-moose]

Is it possible that the win installation has gotten hosed. Do you have an old backup (I see you use a raw drive) that you could put down and see if it works?

[grooveman]

I do....

I'm now operating under the assumption that it is a windows problem (instead of Qemu/KVM)... and trying to do some basic health fixes. I started with a chkdsk. Because the system to be checked is the system and boot drive, however, it forced me to restart, since it cannot unmount it while it is running. Upon restarting, it tells me it is going to do a check, and begins a count down from 10 to.... well, 6. At which point, it just boots the OS as normal. I cannot get the thing to do a chkdsk. Everytime the counter hits six, it boots normally. So, I have the copy of the the image on a USB drive that I've been using to trouble shoot. I added my normal windows drive as a second hard drive to my config, and booted. It came up (very slowly of course), and my drive came up in diskmanager as "Offline". So I put it online, and it came up as drive F:. Great. Then I open a command prompt and do the chckdsk. It unmounts the drive and does its thing. There is no real error output enumerated, but it tells me that it fixed a minor issue with the free space. Cool. Maybe that's it. So I go back to my original config, reboot -- and the thing is totally hosed. The hard drive is not detected at all... Not even in a recovery environment offered in a win10 installation disk. I cannot even do a bcdedit or a bootrec /fixmbr... because the drive isn't detected.

I've never heard of a chkdsk breaking a disk before... so this morning I rebooted to my USB copy of my windows system, with my normal windows system attached, the drive came up offline again, but upon putting it online, it is detected as F:, and I can browse the contents.

So, I'm trying again to see if I can fix it... if not, I'll do a restore.. or maybe a reinstall... Of course, this may not even be my problem at all... I'm still shooting in the dark here... but that is where I'm at...

[grooveman]

Update: It occurred to me that the boot environment probably cannot find the disk because I am using Virtio drivers... I set it to ide, and it was found.

I then was able to fix the boot record.. but I lost my domain trust... so I'm going to have to rejoin and rebuild my profile... I hope it is all worth it. Running a sfc now...

[grooveman]

OK. Here is what I know so far...

I have stripped my original win10 machine down to nothing... it still crashed.

I have checked and rechecked memory and hard drives for any issues. All is good. I moved my image to new media... still crashes.

Finally, I have installed an entirely new system. It worked fine -- until I started adding the Virtio/spice-agent/qemu-agent stuff. I believe that one of these is the culprit... the drivers are incompatible... the services are locking up... something. I boot the new windows system to safe mode, and it will stay up. So it has to be a driver or service...

I'm still trying to isolate it... but I now am 99% certain that the Windows 10 Spring creators is no longer compatible with the virtio/agent stuff that is currently released...

[Anon-E-moose]

I certainly don't have any input into the problem (I personally despise win10) BUT I'm glad to see you're making progress on the potential problem.

Might be worth shooting a report/email at the qemu/virtio devs to see if they have any thoughts on solutions.

[grooveman]

Man... I've tried everything with this. I've spent the last two months workign on this, and tried several reinstalls of windows. I thought I had this licked at least 2 dozen times in the interim.

My last iteration, I installed Windows 10 1809... Let it sit for 24 hours, foudn it to be stable, then added antivirus... waited... then added office... found it to be stable for 24 hours... added our ERP... found it to be stable for 24 hours... etc. etc. I proceeded like that, adding only one package at a time to the system... trying to see where it would break. I did the same with my config... gradually going from low-performance defaults to and eventually winding up with virtio/qxl drivers. I was stable for a full week...

I came in this morning, the system was still up after the weekend. I started going through emails... I turned to talk to one of my staff for about 5 minutes, turned back, and it was locked up... and we are back to where I started... it locks up, usually when it isn't being used.. and there is NOTHING in the event log or linux logs. QEMU remains up... but windows locks up... not reachable via rdp... not pingable.

I've never put forth more effort into trouble-shooting something that this. Ever... and I still have nothing to go on. I really thought after 1 week of solid up-time, that I was golden, and I thought that the fresh install of 1809 was the key... nope. It has crashed 3 times in the last 2 hours.

[Anon-E-moose]

So a question, does qemu behave itself (for a while) if linux is rebooted or fresh started before you fire it up?

[grooveman]

Yes. Qemu itself is rock-solid. It is the Windows VM that rests upon it that keeps going out on me... I've tried at least 4 versions of Qemu... Qemu itself never seems to be the problem.

I just rolled back from the virtio disk driver (back to AHCI)... that was the very last thing I changed last Friday a.m. (today being Monday). The virtio driver was suspect from the beginning... under my old windows install, removing wasn't enough. We'll see how it is with 1809 and the AHCI combination...

[Chiitoo]

What USE-flags is 'app-emulation/qemu' built with?

I see you have 'gtk' enabled globally, and I believe it will be used as the default display type if available (though I'm not entirely sure), at least in recent versions of QEMU.

If it is indeed being used, have you tried, for example, 'sdl' instead?

Likewise I see 'nvidia' in your '--info'. Which version of 'x11-drivers/nvidia-drivers' are in use?

I've been seeing something similar, though it appears that my Linux VMs haven't actually locked up, only the display isn't being refreshed, and they take no mouse or keyboard inputs. I noticed that there was still a bunch of CPU activity in there, and that some emerges had finished up regardless, so I tried to ssh to one of them which was indeed successful.

I do believe it has been happening with a Windows install I did recently as well, but I haven't tried any form of 'ssh' connection to it.

It seems like it's not happening when I use '-display sdl', but it could still be too early to say even after several days due to the random nature of it all...

[grooveman]

Hi Chiltoo,

A fair question...

Currently it is built as such:

Code: Select all

 [ebuild   R   ~] app-emulation/qemu-3.0.0::gentoo  USE="aio alsa bluetooth bzip2 caps curl fdt filecaps jpeg ncurses nls opengl pin-upstream-blobs png pulseaudio seccomp spice usb vhost-net xattr -accessibility -capstone -debug -glusterfs -gnutls -gtk -gtk2 -infiniband -iscsi -lzo -nfs -numa -python -rbd -sasl -sdl -sdl2 (-selinux) -smartcard -snappy -ssh (-static) -static-user -systemtap -tci -test -usbredir -vde -virgl -virtfs -vnc -vte -xen -xfs" PYTHON_TARGETS="python2_7 python3_6 -python3_4 -python3_5" QEMU_SOFTMMU_TARGETS="i386 x86_64 -aarch64 -alpha -arm -cris -hppa -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -ppcemb -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="i386 x86_64 -aarch64 -aarch64_be -alpha -arm -armeb -cris -hppa -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -xtensa -xtensaeb" 0 KiB

I really didn't want to use sdl, however, because I want to use spice (better performance, and multi-monitor support)

I have rolled my system back to SATA emulation for the hard drive, and I've been stable for about 2 hours now... I'm wondering if the win 10 1809 + SATA emulation might be the ticket... I want to give this a fair try. If that doesn't work, I'll keep your suggestion in my back pocket for trouble-shooting purposes (though I really need the multi-monitor support to make this useful).

I'll post back my results. Thanks.

G

[grooveman]

Well, this is interesting...

Since i switched back to the SATA emulation on my Win 10 1809 system, I have had no issues with stability... So I think I've come full-circle.

Maybe this is the silver bullet... or even silver bullets... 1809 and SATA emulation. If it survives the night... I think I will kill this thread.

[grooveman]

Still on the air.

That clinches it for me. The Hard drive virtio disk drivers lost compatbility with win 10 at the 1803 release. It has to be.

I guess at this point, we wait for another release... I don't know what will happen there, now that redhat is owned by IBM, and redhat were the ones kind enough to release these drivers. I don't know if Fedora will pick that up or not...

At least the virtio network drivers are still working...