Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
GPU Passthrough Xen / KVM - VFIO issues - please advice
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 199

PostPosted: Wed Dec 13, 2017 8:53 pm    Post subject: GPU Passthrough Xen / KVM - VFIO issues - please advice Reply with quote

Hi,

Once my new CPU chip arrives I want to start pass my GPU to a Windows VM. Does someone have experiences which solution (Xen, KVM/QEMU) will offer better performance, easier setup/maintenance, less issues, more secure? Or knows any reference?

I have experience with Windows clients in both solutions and I believe KVM/QEMU with SPICE offers slightly better graphic performance when the graphic card is virtualized, but I might be wrong because I didn't fully compare it.


When there are virtualized gamers/people with needed strong gpu in Windows for other purposes here I am happy to reading your experiences.

Probably when my new equipment arrives before Christmas I will do some little research for this and will share the results here.
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen


Last edited by Spargeltarzan on Sat Dec 30, 2017 4:29 pm; edited 1 time in total
Back to top
View user's profile Send private message
littletux
n00b
n00b


Joined: 08 Dec 2003
Posts: 59

PostPosted: Fri Dec 22, 2017 12:17 am    Post subject: Reply with quote

As I know, the feature to share a graphics card to the host and guest system at the same time isn't not fully functional and implemented in kernel, so you need 2 graphics cards, one for the guest and one for the host, because if you haven't this, in the moment you start a guest with a passthrough activated graphics, the host will crash.

I have seen a configuration from a friend, with qemu/kvm and the performance is very good.
Back to top
View user's profile Send private message
fluffysheap
n00b
n00b


Joined: 10 Dec 2017
Posts: 3

PostPosted: Fri Dec 29, 2017 10:17 am    Post subject: Reply with quote

I have such a configuration, which I use to play games in Windows. It's quite a bit better than WINE. WINE barely runs any of the games I wanted to play. I use KVM with an AMD video card. I don't use SPICE. It provides no useful features for me.

The main requirement is that you must have a CPU with support for both VT-D and VT-X (or the AMD equivalents). VT-D is what you need for I/O virtualization, VT-X is for CPU virtualization. Just about everything has VT-X nowadays, but VT-D is a higher-end feature mostly found on i7/i9 and Xeon chips. Even if you have that, there can be some complications with getting the right PCI devices passed through to Windows. The kernel has to be able to isolate a device from the rest of the system in order to pass it through. I was lucky, and my system configuration was fully supported. But there are kernel patches you might need if your motherboard doesn't do things the right way. Read about IOMMU groups for more information. You should also have a UEFI-compatible video card. Most everything modern supports this (I'm red team, so nearly any R7, R9 or newer card will work - as will some of the HD 7xxx series). You don't need a UEFI-compatible motherboard, though - the UEFI for the guest is provided by the VM software!

I use KVM and libvirt to manage the VM. Everything can be done with official Gentoo software in Portage. Though it's called libvirt, it's really a command line tool, not a library. Additionally you can use the virt-manager GUI package. This is a good way to get started but it is not sufficient for everything you'll likely need to do. You will need to become familiar with the virsh tool and its XML data format to define VMs. For performance, it's much better to have keyboard and mouse passed through to Windows as hardware, rather than through the software tunnel of the GUI. Once you have that, the GUI is just a fancy launcher that gets in the way.

Even if you read a guide that says you should, don't use the emulated Q35 motherboard. Stick with the i440FX. Support for Q35 is very poor. Using the virtual i440FX will probably not cost you any features.

While all the guides say you need two video cards, you don't really, although it does make setting up easier as otherwise you can't use the GUI and might have to log in from another PC. I used a DisplayLink USB display adapter, since I have a 3930K CPU which doesn't have an onboard GPU. I found this much easier to set up than all the guides that talk about dual video cards. Now that everything works well, I just quit out of X, run Windows VM from the command line, and it looks just like a native Windows installation, and I don't bother with the extra display adapter any more. Supposedly it's possible to hot-switch back and forth between X and Windows without even stopping the X server, but I haven't bothered. For this, you likely need two keyboards, one to pass through to Windows and one to control Linux.

I would also recommend, if you can, passing through a whole USB host PCI device instead of individual USB devices. If you pass through the host adapter, every device connected to it will be passed through to Windows. This way you also get hotplug support and better reliability. I measured the performance (in terms of latency for mouse/controller lag) and it's the same within the tolerances of my 60FPS video camera, but occasionally I have a glitch with a passed through USB device and have to reset Windows to get it to work again. I have a USB 3.0 PCIe card that I passed through and it works better. Passing through motherboard-based USB ports likely won't work because they won't be properly isolated from the rest of the integrated devices on the motherboard. But you can still pass the individual USB devices connected to them.

For sound I just pass through a USB headset, which I would be using for games anyway. Much easier than dealing with the sound passthrough support. Can use all native Windows drivers, 3D sound, microphone, the works. You could pass through a PCI sound card just as well if you have one, but probably not motherboard-based audio. If you want to use motherboard audio, you'll need to do the sound passthrough to ALSA.

I recommend Windows 10 as a client OS. Windows 7 and 8 can be made to work, but installation is painful, because they don't have very good UEFI support (Windows 7 doesn't have any). I had no success migrating my existing Windows 8 installation into the VM. I just started over with a fresh W10 install. Windows 10 privacy issues are mitigated by the VM. Just pretend it's an XBox that lets you use a keyboard and mouse, and do all your real stuff in Linux. Now Microsoft can't spy on you... much.

I also recommend keeping your OS on a small virtual drive and all your games on a bigger, different one. Now when you reinstall your OS (I think I reinstalled five times while setting up) you don't have to reinstall/redownload all your games. This is another advantage of Windows 10. You can install it without a product key, and only put your key in once you are sure you are happy with the setup.

Some of the guides talk about doing this whole dance where you have to tell the kernel to hot-unplug your GPU, unbind it from the display driver, rebind it to the VFIO driver and all this. I did this at first but then I discovered that none of it is necessary. You need to have VFIO modules built for your kernel, but beyond that all I do is run "virsh start" and everything works.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 199

PostPosted: Sat Dec 30, 2017 4:14 pm    Post subject: Reply with quote

fluffysheap,

Thank you for your very detailed advice! It came exactly on the day where my new RAM and CPU with support for vt-d arrived. Fantastic!

I started to work through Gentoo Passthrough Guide and Archlinux Passthrough Guide and I struggle with fully hidding my GPU and enabling the vfio device.

1) My Mainboard has a CPU based PCIe slot, what is not a problem according to the Archlinux Wiki; I only should be careful not to plugin another card on a second shared PCIe which would be passed through as well because they are in a shared IOMMU_GROUP. (I added iommu to grub.cfg accordingly, and IOMMU can be detected in dmesg)

2) In my IOMMU_GROUP 1 only the PCI bridge, my GPU and its HDMI Audio is listed.
Code:

IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon
IOMMU Group 1 01:00.0 AMD/ATI VGA compatible controller
IOMMU Group 1 01:00.1 Audio device


3) The VFIO script from Gentoo Wiki causes my system to freeze, that's why I hide it manually according to Archlinux Wiki in /etc/modprobe.d/vfio.conf and I added MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...) to /etc/conf.d/modules.conf. I blacklisted radeon

4) According to Archlinux Wiki:
Code:

$ dmesg | grep -i vfio

[    0.329224] VFIO - User Level meta-driver version: 0.3
[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
-----> [    2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103) <-----

My two devices (gpu and hdmi audio) become added, but I miss the last line "enabling device" in dmesg output. Could it have something todo that Intel driver probably is not activated as a module, but compiled in Kernel? (I should check, but don't find the Option currently)

5)
Code:

lspci -nnk -d


My gpu has vfio-pci as a driver, but not the audio card. This is still snd_hda_intel. I tried to hide snd_hda_intel too without success. My mainboard onboard audio uses snd_hda_intel too. In the meantime audio in host doesn't work any more. (Could I even blacklist this driver and still use audio in my Linux host mainboard controller?)

6) I installed Win 8.1 in a UEFI virt-manager machine (will upgrade to Win 10 if I have problems like you wrote, I had a Win8.1 in my hands today). When I passthrough both devices, the machine can't be started due to "no permission".

Any idea what I can do to successfully "enabling vfio-pci" (this last line of the dmesg output - if necessary at all, saw success report without this line - and how to launch my Win virtual machine? Thank you!
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
fluffysheap
n00b
n00b


Joined: 10 Dec 2017
Posts: 3

PostPosted: Sat Dec 30, 2017 11:49 pm    Post subject: Reply with quote

There is a lot of trial and error involved in setting this up. I tried many things before getting it to work.

1) Everything has CPU-based PCIe now, that's fine. You also don't HAVE to pass through all devices on the same IOMMU group - you just can't use them in the host at the same time, so they still need to be unbound from any drivers that claim them. If you do want to pass through another PCIe device, plug it into the other slot in the IOMMU group (if you have a Crossfire motherboard, the Crossfire slots usually go together).

2) In my configuration there is not even a PCI bridge on that IOMMU group, just the GPU and its audio device. But according to others, this is still OK. So you might be OK here. Be sure not to pass the PCI bridge to Windows, though!

But you can still try the ACS override patch if you like (I wouldn't try it first, I think your sound driver is the problem). The patch is here:
https://github.com/zman0900/linux-vfio-aur/blob/master/override_for_missing_acs_capabilities.patch

I haven't built this in a while since my system doesn't need it, but I tried it with 4.9-ish kernel and I don't think I had any trouble applying it.

More information is here: http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html

There is a lot of information on that site about how VFIO and virtualization work. This might solve your problem, but maybe not.

3) VFIO script? That sounds like a problem waiting to happen. What does it do? If you can find out where it freezes, it might point to where your real problem is.

I don't blacklist any modules. I do load VFIO, but I can have VFIO and radeon loaded at the same time, and I don't have any problems. Probably this module dancing was necessary once, but I suspect the radeon module is just able now to deal with the card managed by VFIO, and you don't have to worry about keeping them separated. You do still need to load VFIO first, though. So I have /etc/conf.d/modules which contains:
modules="vfio vfio_iommu_type1 vfio_pci vfio_virqfd" (obviously you should just add these to any other modules= line you may already have). This causes the modules to be loaded before other modules are detected.

However, blacklisting radeon isn't a bad way to make sure that you have no conflicts while you are setting up.

An excerpt of the script I once used to do the song-and-dance is here:

echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove
echo 1 > /sys/bus/pci/devices/0000\:01\:00.1/remove (change these to the PCI bus addresses for your GPU)
modprobe -r radeon
/etc/init.d/udev stop
echo 1 > /sys/bus/pci/rescan
modprobe vfio-pci ids=xxxx:yyyy,zzzz:qqqq (PCI IDs of the GPU and its audio)
modprobe vfio_iommu_type1

I don't think it's necessary but I don't think it hurts anything either.

I have only the following in my /etc/modprobe.d/vfio.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm ignore_msrs=1

And I'm not sure I even need those, but I tried them when I was dealing with some Windows problems. They don't seem to hurt.

In general I think most of the guides were written at a time when VFIO was a lot fussier than it is now, and you can pretty much just let it do its thing now.

4) I don't use my sound card HDMI audio. If you don't use it either, I recommend just disabling all linux drivers associated with it. HDMI audio would be a way to get native sound in Windows, though, if you were interested in that.

5) This could easily be your problem. You must have all drivers for your passthrough devices unbound. I would recompile with your sound driver as a module, but you could also probably prevent it from binding to the GPU audio on the kernel command line somehow. The kernel config option is CONFIG_SND_HDA_INTEL. Rather than look around for hours in menuconfig, edit kernel .config file with a text editor and find the line with this option with text search. If it's set to y, then delete the line and run "make oldconfig." It should let you change it to a module.

In my case, I don't use either the GPU audio OR the motherboard audio. I have a PCI sound card that I use in Linux, and a USB headset that I use in Windows. So I don't actually know whether this is even the right driver for GPU HDMI audio. But if lspci says it's bound (to anything other than vfio or pci-stub) then it's definitely going to prevent you passing through the device.

Sound is a minor thing compared to the GPU passthrough so my instinct is to just get it out of the way and focus on the important parts.

In the short term I would recommend just blacklisting that module so it stays out of your way. You won't have sound in Linux during this time. You can work on getting it re-enabled in Linux once you have KVM working. You will probably just need to pass the right options in modules.conf or /etc/udev/rules.d so it doesn't interfere with your HDMI audio. If you want to actually use the HDMI audio in linux, you'll probably have to use something like my song and dance script to reorganize the module binding whenever you start or stop the VM.

6) Once installed, Windows 8 should continue to work. I start the VM with sudo. It's really just as simple as sudo virsh start <name of vm>. If you still have "permission" errors as root, then it's probably actually a hardware issue. Probably installation was easier for you because I was working with an 8.0 install disc, I think UEFI works better in 8.1.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum