QEMU KVM GPU passthrough – amdgpu grabs card before vfio-pci

[sahnow]

Hello all. I’m attempting to get my RX6700XT GPU to pass through to a libvirt QEMU KVM Windows VM. I have a Ryzen 9 7950X processor, whose integrated graphics I wish to use for the Gentoo host.

I’m fairly certain I’ve enabled IOMMU using the wiki article; however, I’ve been unsuccessful in making my RX6700XT use the vfio-pci kernel module instead of amdgpu.

IOMMU

Code: Select all

/etc/default/grub

GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on"

Code: Select all

~ $ dmesg | grep AMD-Vi
[    0.048307] AMD-Vi: Unknown option - 'on'
[    0.113330] AMD-Vi: Using global IVHD EFR:0x246577efa2254afa, EFR2:0x0
[    2.417037] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    2.417703] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.417706] AMD-Vi: Extended features (0x246577efa2254afa, 0x0): PPR NX GT [5] IA GA PC GA_vAPIC
[    2.417713] AMD-Vi: Interrupt remapping enabled
[    2.459769] AMD-Vi: Virtual APIC enabled

I turned the option on in the BIOS too but is IOMMU really enabled? It never says “IOMMU enabled”, like the wiki’s command output. I realize that example is for an Intel system and mine will look different, but this output seems ambiguous to me nonetheless.

VFIO

Attempt #1

Code: Select all

~ $ lspci -nn
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] [1002:73df] (rev c5)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

Code: Select all

/etc/modprobe.d/vfio

options vfio-pci ids=1002:73df,1002:ab28

Code: Select all

/etc/conf.d/modules

modules="vfio-pci"

The wiki says to add the IDs just like that. In other guides, I’ve seen recommendations to add vfio, vfio_iommu_type1, and vfio_virqfd to OpenRC’s list of kernel modules. Adding the first two gave me warnings that they were already loaded and the latter one seems to be included in the kernel these days so I’ve only added vfio-pci. When I restart my computer however, I see this output at the end and it hangs.

Code: Select all

* Loading kernel modules ...
insmod /lib/modules/6.6.28-gentoo-dist/kernel/drivers/iommu/iommufd/iommufd.ko
insmod /lib/modules/6.6.28-gentoo-dist/kernel/drivers/vfio/vfio.ko
insmod /lib/modules/6.6.28-gentoo-dist/kernel/drivers/vfio/vfio_iommu_type1.ko
insmod /lib/modules/6.6.28-gentoo-dist/kernel/drivers/vfio/pci/vfio-pci-core.ko
insmod /lib/modules/6.6.28-gentoo-dist/kernel/drivers/vfio/pci/vfio-pci.ko ids=1002:73df,1002:ab28

Attempt #2

Code: Select all

/etc/default/grub

GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on vfio-pci.ids=1002:73df,1002:ab28"

Code: Select all

/etc/modprobe.d/amdgpu.conf

softdep amdgpu pre: vfio-pci
# softdep amd_iommu_v2 pre: vfio-pci

Code: Select all

/etc/conf.d/modules

modules="vfio-pci"

I tried several combinations of setting the vfio-pci IDs in GRUB vs. /etc/modprobe.d/vfio.conf and one or both of the lines in /etc/modprobe.d/amdgpu.conf, all to no avail. I’m forced to chroot in and remove them to be able to finish booting again.

Setting only the above line in /etc/default/grub allows me to finish booting, though sometimes on the second or third restart. However, once I’m logged in, I can see that amdgpu has taken over the card.

Code: Select all

~ $ lspci -knn
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] [1002:73df] (rev c5)
  Subsystem: ASRock Incorporation Navi 22 [Radeon RX 6700/6700 XT/6750XT / 6800M/6850M XT] [1849:5210]
  Kernel driver in use: amdgpu
  Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
  Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/21 HDMI/DP Audio Controller [1002:ab28]
  Kernel driver in use: snd_hda_intel
  Kernel modules: snd_hda_intel

The modules are installed. modinfo returns output for vfio, vfio-pci, vfio_iommu_type1, amdgpu, but not vfio_virqfd (which shouldn’t be a problem) or amd_iommu_v2 (which probably means that commented line in /etc/modprobe.d/amdgpu is superfluous).

I would love to be pointed in the right direction! And by all means, let me know if I need to include some more information.

Thank you for your consideration.