Qemu/KVM unstable

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Hello.

I've had this problem now for over 2 months. I'm not sure if it is an issue with the kernel, qemu, or windows, but I'm having some pretty serious stability issues with my qemu win10 machine.

Because I wear a lot of hats at work, I have to have a windows system. I've been successfully using qemu/kvm for this for years. About 2 months ago I did an update, and found that my windows vm has been crashing regularly ever since. But, every day since then, without fail, my windows system has crashed multiple times.

Typically, it locks up when idle, though twice it just froze in the middle of writing emails or viewing an excel document. Sometimes it locks up 2 minutes after going idle, and other times is locks up 6 hours after going idle -- there doesn't seem to be any rhyme or reason to it. Furthermore, there doesn't seem to be any really useful information in the windows Event viewer, the qemu console, or on the Gentoo host system. I do get some chatter in /var/log/messages that might be related:

LIsLinuxIsSogood · Veteran Joined: 13 Feb 2016 Posts: 1179

grooveman, hey it's been a while since my trying to use qemu to boot from a physical disk, is that what you are doing here? anyway it seems due to your explanation of what's been tried that you pretty much can rule out the hard drive and kernel for now. (I would suggest something, but not sure if it is going to make a bit of difference)...In theory at least the command for starting qemu seems very specific and probably involves a lot of settings that the guest windows host won't agree with. At least my experience with virtual machines is that over time they do seem to break/die/fall apart a bit faster than their physical machine counterparts. Could you try modifying your launch command to see if other QEMU START COMMANDS WOULD HAVE ANY IMPACT ON THIS SITUATION??!

I don't actually know what command to pass, but it looks like you must have spent some time figuring out the exact start command for your system, or else you found it somewhere, and then modified that based on your custom needs for the command.

Could it be the mem or cpu settings on you qemu start command, apologies if this isn't as helpful, but wow look at that call trace being 20 lines!!

EDIT: Have you also attempted to send the log from the entire virtual machine startup process "captured" into a file somewhere? Then that could actually be informative as to whether or not the fix may involve something more permanent like a need to be doing a reinstallation of the guest...and hopefully not.)

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Ok.. I've ruled out bad RAM as an issue...

And I've reduced my qemu command to the barest of essentials:

Anon-E-moose · Posted: Mon Nov 05, 2018 4:10 pm Post subject:

Have you tried adding "-D <logfile>" to see if it will leave a hint?

ETA: man qemu and look for logfile to get several different options.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

I can certainly do that... but my understanding is that just takes what would have otherwise been sent to standard error... which I have been monitoring.

Can't hurt though... I"ll add it to the command. I appreciate the input.
_________________
To look without without looking within is like looking without without looking at all.

Anon-E-moose · Posted: Mon Nov 05, 2018 4:27 pm Post subject:

There are some debug options available depending on what you think might be causing the problem.
That's why I mentioned the man page.

ETA: Also https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems, even though it's RH, it might give a clue or two.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Anon-E-moose · Posted: Tue Nov 06, 2018 6:11 pm Post subject:

which version of qemu? Have you updated it lately, at least since the last time it worked?
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

The problem has occurred ever since an update about 2.5 months ago now. I've tried every version of Qemu in portage during this time, and have updated numerous times between then and now (as well as every kernel update)... hoping that that some bug or incompatibility that might be responsible for this would be rectified... But this has not happened. I'm currently running Qemu 3.0, but it doesn't really seem to matter... I've stripped my command to the bare essentials...and it still does it.

I don't see how it could be something I'm doing... I was really hoping that someone was a aware of a new virtual driver for win 10, or a kernel patch, or new version of Qmeu or a new switch to qmeu that would resolve it...

I'll keep banging away at it...

Thanks.
_________________
To look without without looking within is like looking without without looking at all.

Anon-E-moose · Posted: Tue Nov 06, 2018 7:39 pm Post subject:

Is it possible that the win installation has gotten hosed. Do you have an old backup (I see you use a raw drive) that you could put down and see if it works?
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

I do....

I'm now operating under the assumption that it is a windows problem (instead of Qemu/KVM)... and trying to do some basic health fixes. I started with a chkdsk. Because the system to be checked is the system and boot drive, however, it forced me to restart, since it cannot unmount it while it is running. Upon restarting, it tells me it is going to do a check, and begins a count down from 10 to.... well, 6. At which point, it just boots the OS as normal. I cannot get the thing to do a chkdsk. Everytime the counter hits six, it boots normally. So, I have the copy of the the image on a USB drive that I've been using to trouble shoot. I added my normal windows drive as a second hard drive to my config, and booted. It came up (very slowly of course), and my drive came up in diskmanager as "Offline". So I put it online, and it came up as drive F:. Great. Then I open a command prompt and do the chckdsk. It unmounts the drive and does its thing. There is no real error output enumerated, but it tells me that it fixed a minor issue with the free space. Cool. Maybe that's it. So I go back to my original config, reboot -- and the thing is totally hosed. The hard drive is not detected at all... Not even in a recovery environment offered in a win10 installation disk. I cannot even do a bcdedit or a bootrec /fixmbr... because the drive isn't detected.

I've never heard of a chkdsk breaking a disk before... so this morning I rebooted to my USB copy of my windows system, with my normal windows system attached, the drive came up offline again, but upon putting it online, it is detected as F:, and I can browse the contents.

So, I'm trying again to see if I can fix it... if not, I'll do a restore.. or maybe a reinstall... Of course, this may not even be my problem at all... I'm still shooting in the dark here... but that is where I'm at...
_________________
To look without without looking within is like looking without without looking at all.

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Update: It occurred to me that the boot environment probably cannot find the disk because I am using Virtio drivers... I set it to ide, and it was found.

I then was able to fix the boot record.. but I lost my domain trust... so I'm going to have to rejoin and rebuild my profile... I hope it is all worth it. Running a sfc now...
_________________
To look without without looking within is like looking without without looking at all.

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

OK. Here is what I know so far...

I have stripped my original win10 machine down to nothing... it still crashed.

I have checked and rechecked memory and hard drives for any issues. All is good. I moved my image to new media... still crashes.

Finally, I have installed an entirely new system. It worked fine -- until I started adding the Virtio/spice-agent/qemu-agent stuff. I believe that one of these is the culprit... the drivers are incompatible... the services are locking up... something. I boot the new windows system to safe mode, and it will stay up. So it has to be a driver or service...

I'm still trying to isolate it... but I now am 99% certain that the Windows 10 Spring creators is no longer compatible with the virtio/agent stuff that is currently released...
_________________
To look without without looking within is like looking without without looking at all.

Anon-E-moose · Posted: Thu Nov 15, 2018 6:32 pm Post subject:

I certainly don't have any input into the problem (I personally despise win10) BUT I'm glad to see you're making progress on the potential problem.

Might be worth shooting a report/email at the qemu/virtio devs to see if they have any thoughts on solutions.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Man... I've tried everything with this. I've spent the last two months workign on this, and tried several reinstalls of windows. I thought I had this licked at least 2 dozen times in the interim.

My last iteration, I installed Windows 10 1809... Let it sit for 24 hours, foudn it to be stable, then added antivirus... waited... then added office... found it to be stable for 24 hours... added our ERP... found it to be stable for 24 hours... etc. etc. I proceeded like that, adding only one package at a time to the system... trying to see where it would break. I did the same with my config... gradually going from low-performance defaults to and eventually winding up with virtio/qxl drivers. I was stable for a full week...

I came in this morning, the system was still up after the weekend. I started going through emails... I turned to talk to one of my staff for about 5 minutes, turned back, and it was locked up... and we are back to where I started... it locks up, usually when it isn't being used.. and there is NOTHING in the event log or linux logs. QEMU remains up... but windows locks up... not reachable via rdp... not pingable.

I've never put forth more effort into trouble-shooting something that this. Ever... and I still have nothing to go on. I really thought after 1 week of solid up-time, that I was golden, and I thought that the fresh install of 1809 was the key... nope. It has crashed 3 times in the last 2 hours.
_________________
To look without without looking within is like looking without without looking at all.

Anon-E-moose · Posted: Mon Dec 03, 2018 3:15 pm Post subject:

So a question, does qemu behave itself (for a while) if linux is rebooted or fresh started before you fire it up?
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Yes. Qemu itself is rock-solid. It is the Windows VM that rests upon it that keeps going out on me... I've tried at least 4 versions of Qemu... Qemu itself never seems to be the problem.

I just rolled back from the virtio disk driver (back to AHCI)... that was the very last thing I changed last Friday a.m. (today being Monday). The virtio driver was suspect from the beginning... under my old windows install, removing wasn't enough. We'll see how it is with 1809 and the AHCI combination...
_________________
To look without without looking within is like looking without without looking at all.

Chiitoo · Posted: Mon Dec 03, 2018 4:37 pm Post subject: ><)))°€

What USE-flags is 'app-emulation/qemu' built with?

I see you have 'gtk' enabled globally, and I believe it will be used as the default display type if available (though I'm not entirely sure), at least in recent versions of QEMU.

If it is indeed being used, have you tried, for example, 'sdl' instead?

Likewise I see 'nvidia' in your '--info'. Which version of 'x11-drivers/nvidia-drivers' are in use?

I've been seeing something similar, though it appears that my Linux VMs haven't actually locked up, only the display isn't being refreshed, and they take no mouse or keyboard inputs. I noticed that there was still a bunch of CPU activity in there, and that some emerges had finished up regardless, so I tried to ssh to one of them which was indeed successful.

I do believe it has been happening with a Windows install I did recently as well, but I haven't tried any form of 'ssh' connection to it.

It seems like it's not happening when I use '-display sdl', but it could still be too early to say even after several days due to the random nature of it all...
_________________
Kindest of regardses.

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Hi Chiltoo,

A fair question...

Currently it is built as such:

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Well, this is interesting...

Since i switched back to the SATA emulation on my Win 10 1809 system, I have had no issues with stability... So I think I've come full-circle.

Maybe this is the silver bullet... or even silver bullets... 1809 and SATA emulation. If it survives the night... I think I will kill this thread.
_________________
To look without without looking within is like looking without without looking at all.

grooveman · Veteran Joined: 24 Feb 2003 Posts: 1217

Still on the air.

That clinches it for me. The Hard drive virtio disk drivers lost compatbility with win 10 at the 1803 release. It has to be.

I guess at this point, we wait for another release... I don't know what will happen there, now that redhat is owned by IBM, and redhat were the ones kind enough to release these drivers. I don't know if Fedora will pick that up or not...

At least the virtio network drivers are still working...
_________________
To look without without looking within is like looking without without looking at all.