QEMU - PCI passthrough [solved]

[Moldmaker]

Hi everybody, (now I got it right, sorry once more)

I am trying to use the very old PCI Fritz ISDN card which is part of my computer to configure my ISDN telephone box. The software needed for this does not work under 64 bit system, and although the ISDN card is being recognized correctly, I can not use it. So I want to use it from a virtualized system. I emerged QEMU and set up an openSUSE 10.3. Now I have to assign the ISDN card to the virtual suse. I found those instructions . First I have to unbind the PCI card from the host kernel driver:

Code: Select all

lspci
.
05:01.0 Network controller: AVM GmbH Fritz!Card PCI v2.0 ISDN (rev 02)
.

and then

Code: Select all

modprobe pci_stub
lspci -n

to see the vendor and device ID:

Code: Select all

.
05:01.0 0280: 1244:0e00 (rev 02)
.

Then I placed this information in the following files:

Code: Select all

echo "1244 0e00" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:01.0" > /sys/bus/pci/devices/0000:05:01.0/driver/unbind
echo "0000:05:01.0" > /sys/bus/pci/drivers/pci-stub/bind

Load the KVM modules:

Code: Select all

modprobe kvm
modprobe kvm-intel

The next step would be to assign the device to the virtual machine. I did (as normal user):

Code: Select all

/usr/bin/qemu-system-x86_64  -soundhw ac97 -machine accel=kvm -m 2048 -cdrom "/dev/sr0" -hda "openSUSE_10.3" -boot once=d,menu=off -net none -rtc base=localtime -device pci-assign,host=05:01.0 -name  "Linux 2.6" $*

and the answer is

Code: Select all

qemu-system-x86_64: pci-assign,host=05:01.0: drive with bus=0, unit=0 (index=0) exists

Then I learned that the device must have a name and tried this one:

Code: Select all

/usr/bin/qemu-system-x86_64  -soundhw ac97 -machine accel=kvm -m 2048 -cdrom "/dev/sr0" -hda "openSUSE_10.3" -boot once=d,menu=off -net none -rtc base=localtime -device fritzcard pci-assign,host=05:01.0 -name "Linux 2.6" $*

and again the answer was

Code: Select all

qemu-system-x86_64: pci-assign,host=05:01.0: drive with bus=0, unit=0 (index=0) exists

How can this be solved? Would appreciate any help.

[alamahant]

Hi
1.You need to add in /etc/default/grub

Code: Select all

GRUB_CMDLINE_LINUX="....intel_iommu=on......." ###for intel cpu or 
GRUB_CMDLINE_LINUX="....intel_iommu=on......" ###for amd 

2.create a file /etc/modprobe.d/vfio.conf

Code: Select all

options vfio-pci ids=1244:0e00

3.make sure that vfio-pci module get loaded @boot

Code: Select all

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

4.Update grub and initramfs
5.Please make sure that your vm boots in uefi(install the ovmf package)
6.In virt-manager assign the specific card to the vm.
Thats it really.

[Moldmaker]

Thank you for your reply. I did everything exactly as you told me to do. (One thing: When I installed libvirt and virt-manager and added the daemon libvirtd to the default runlevel, X doesn't start anymore. So I start it mnually after logging in.) I created a new VM and installed opensuse 10.3 in it. (If you ask yourself why this distro: It has full ISDN functionality.)

5.Please make sure that your vm boots in uefi(install the ovmf package)

It was already installed before, how can I check that the VM boots in UEFI?

6.In virt-manager assign the specific card to the vm.

I did. Then I started the VM and get this:

Code: Select all

Fehler beim Starten der Domain: Nicht unterstützte Konfiguration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1400, in startup
    self._backend.create()
  File "/usr/lib/python3.7/site-packages/libvirt.py", line 1089, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: Nicht unterstützte Konfiguration: host doesn't support passthrough of host PCI devices

How can I solve this?

[alamahant]

Hi if you have already created your VM using bios there is nothing you can do.
Just reinstall your VM and this time in virt manager VM settings choose uefi.
You have to have installed

Code: Select all

sys-firmware/edk2-ovmf

Of cource you need to have your user member of the libvirt group

Code: Select all

usermod -aG libvirt $USER

Please also check this

Code: Select all

dmesg | grep -E "DMAR|IOMMU"

to see if iommu is indeed enabled.
I think passthrough will NOT work until you create your uefi VM.
NOTE you need to also update initramfs and grub when you are done.
This is how I do it for GPU passthrough and it all works flawlessly.

[Moldmaker]

Thank you again. I will do as you said. Additionally I found something else I missed. There is a tool named virt-host-validate which told me to pass the argument

Code: Select all

intel_iommu=on

to the kernel when booting. I did and now everything looks good:

Code: Select all

# virt-host-validate
  QEMU: Überprüfung for hardware virtualization                                 : BESTANDEN
  QEMU: Überprüfung if device /dev/kvm exists                                   : BESTANDEN
  QEMU: Überprüfung if device /dev/kvm is accessible                            : BESTANDEN
  QEMU: Überprüfung if device /dev/vhost-net exists                             : BESTANDEN
  QEMU: Überprüfung if device /dev/net/tun exists                               : BESTANDEN
  QEMU: Überprüfung for cgroup 'cpu' controller support                         : BESTANDEN
  QEMU: Überprüfung for cgroup 'cpuacct' controller support                     : BESTANDEN
  QEMU: Überprüfung for cgroup 'cpuset' controller support                      : BESTANDEN
  QEMU: Überprüfung for cgroup 'memory' controller support                      : BESTANDEN
  QEMU: Überprüfung for cgroup 'devices' controller support                     : BESTANDEN
  QEMU: Überprüfung for cgroup 'blkio' controller support                       : BESTANDEN
  QEMU: Überprüfung for device assignment IOMMU support                         : BESTANDEN
  QEMU: Überprüfung if IOMMU is enabled by kernel                               : BESTANDEN

BESTANDEN is german for PASSED. However, now I get another error from virt-manager when starting the VM:

Code: Select all

Fehler beim Starten der Domain: Interner Fehler: process exited while connecting to monitor: 2020-06-12T10:04:17.585481Z qemu-system-x86_64: -device vfio-pci,host=05:01.0,id=hostdev0,bus=pci.0,addr=0x8: vfio 0000:05:01.0: failed to setup container for group 9: Failed to set iommu for container: Operation not permitted

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1400, in startup
    self._backend.create()
  File "/usr/lib/python3.7/site-packages/libvirt.py", line 1089, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: Interner Fehler: process exited while connecting to monitor: 2020-06-12T10:04:17.585481Z qemu-system-x86_64: -device vfio-pci,host=05:01.0,id=hostdev0,bus=pci.0,addr=0x8: vfio 0000:05:01.0: failed to setup container for group 9: Failed to set iommu for container: Operation not permitted 

This might be related to what you said before, so I am going to check it and get back to this thread.

[Moldmaker]

You have to have installed

Code: Select all

sys-firmware/edk2-ovmf

Was installed before.

Of cource you need to have your user member of the libvirt group

Code: Select all

usermod -aG libvirt $USER

Was not set. Now it is.

Please also check this

Code: Select all

dmesg | grep -E "DMAR|IOMMU"

to see if iommu is indeed enabled.

Code: Select all

# dmesg | grep -E "DMAR|IOMMU"
[    0.021791] ACPI: DMAR 0x00000000CBE900C0 000130 (v01 AMI    OEMDMAR  00000001 MSFT 00000097)
[    0.245187] DMAR: IOMMU enabled
[    0.290217] DMAR-IR: This system BIOS has enabled interrupt remapping
[    1.213801] DMAR: Host address width 39
[    1.213804] DMAR: DRHD base: 0x000000fbfff000 flags: 0x0
[    1.213821] DMAR: dmar0: reg_base_addr fbfff000 ver 1:0 cap c9008010e60262 ecap f020fa
[    1.213823] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    1.213829] DMAR: dmar1: reg_base_addr fbffe000 ver 1:0 cap c9078010ef0462 ecap f020fe
[    1.213832] DMAR: RMRR base: 0x000000000e4000 end: 0x000000000e87ff
[    1.213835] DMAR: RMRR base: 0x000000cbeeb800 end: 0x000000cbefffff
[    1.213837] DMAR: ATSR flags: 0x0
[    1.213941] DMAR: dmar0: Using Queued invalidation
[    1.213952] DMAR: dmar1: Using Queued invalidation
[    1.213969] Your BIOS is broken; DMA routed to ISOCH DMAR unit but no TLB space.
[    1.214125] DMAR: Hardware identity mapping for device 0000:00:1b.0
[    1.214134] DMAR: Setting RMRR:
[    1.214208] DMAR: Setting identity map for device 0000:00:1a.0 [0xcbeeb800 - 0xcbefffff]
[    1.214315] DMAR: Setting identity map for device 0000:00:1a.1 [0xcbeeb800 - 0xcbefffff]
[    1.214399] DMAR: Setting identity map for device 0000:00:1a.2 [0xcbeeb800 - 0xcbefffff]
[    1.214477] DMAR: Setting identity map for device 0000:00:1a.7 [0xcbeeb800 - 0xcbefffff]
[    1.214563] DMAR: Setting identity map for device 0000:00:1d.0 [0xcbeeb800 - 0xcbefffff]
[    1.214646] DMAR: Setting identity map for device 0000:00:1d.1 [0xcbeeb800 - 0xcbefffff]
[    1.214734] DMAR: Setting identity map for device 0000:00:1d.2 [0xcbeeb800 - 0xcbefffff]
[    1.214817] DMAR: Setting identity map for device 0000:00:1d.7 [0xcbeeb800 - 0xcbefffff]
[    1.214833] DMAR: Setting identity map for device 0000:00:1a.0 [0xe4000 - 0xe87ff]
[    1.214843] DMAR: Setting identity map for device 0000:00:1a.1 [0xe4000 - 0xe87ff]
[    1.214854] DMAR: Setting identity map for device 0000:00:1a.2 [0xe4000 - 0xe87ff]
[    1.214865] DMAR: Setting identity map for device 0000:00:1a.7 [0xe4000 - 0xe87ff]
[    1.214876] DMAR: Setting identity map for device 0000:00:1d.0 [0xe4000 - 0xe87ff]
[    1.214886] DMAR: Setting identity map for device 0000:00:1d.1 [0xe4000 - 0xe87ff]
[    1.214897] DMAR: Setting identity map for device 0000:00:1d.2 [0xe4000 - 0xe87ff]
[    1.214908] DMAR: Setting identity map for device 0000:00:1d.7 [0xe4000 - 0xe87ff]
[    1.214920] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.214991] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.215125] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.682014] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    1.682968] AMD IOMMUv2 functionality not available on this system
[  216.214094] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

NOTE you need to also update initramfs and grub when you are done.

This I did right ...

[alamahant]

I had also advised you
to add intel_iommu=on to grub..
You most have probaly forgotten to run

Code: Select all

grub-mkconfig -o /boot/grub/grub.cfg

Also you need to update initramfs.
with dracut I do it thus

Code: Select all

dracut --force

I do not know what if any initramfs utility you are using....

[Moldmaker]

I think I put it all together now. But still it does not work. I have learned that qemu needs to know about the location of the OVMF files, and I found your post here in the forum. I followed the instructions but still the UEFI option is inactive (UEFI not found) when I create a new VM. Maybe there is anything else I missed?

[alamahant]

I am out of ideas......
Normally uefi should not have been greyed out nowadays even without the thread you mentioned.
For me it works nicely..

[Moldmaker]

I am out of ideas......
Normally uefi should not have been greyed out nowadays even without the thread you mentioned.

Are these versions the same you have installed?

Code: Select all

app-emulation/qemu-4.0.0-r3::nokit was built with the following:
USE="aio alsa bzip2 caps curl fdt filecaps gtk jpeg ncurses nls opengl pin-upstream-blobs png pulseaudio python sdl seccomp spice usb usbredir vhost-net vnc xattr -accessibility -capstone -debug -doc -glusterfs -gnutls -infiniband -iscsi -lzo -nfs -numa -rbd -sasl (-selinux) -smartcard -snappy -ssh -static -static-user -systemtap -tci -test -vde -virgl -virtfs -vte -xen -xfs" PYTHON_TARGETS="python2_7 python3_7" QEMU_SOFTMMU_TARGETS="i386 x86_64 -aarch64 -alpha -arm -cris -hppa -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="i386 x86_64 -aarch64 -aarch64_be -alpha -arm -armeb -cris -hppa -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -xtensa -xtensaeb"

>>> Attempting to run pkg_info() for 'app-emulation/qemu-4.0.0-r3'
Using:
  app-emulation/spice-protocol-0.12.14
  sys-firmware/edk2-ovmf-2017_p20180211
    USE=binary
  sys-firmware/ipxe-1.0.0_p20180211
  sys-firmware/seabios-1.11.0-r1
    USE=binary
  sys-firmware/sgabios-0.1_pre8-r1

Code: Select all

app-emulation/libvirt-5.2.0-r2::nokit was built with the following:
USE="caps dbus libvirtd macvtap nls policykit qemu udev vepa virt-network zeroconf -apparmor -audit -firewalld -fuse -glusterfs -iscsi -iscsi-direct -libssh -lvm -lxc -nfs -numa (-openvz) -parted -pcap -phyp -rbd -sasl (-selinux) -virtualbox -wireshark-plugins -xen -zfs"

Code: Select all

app-emulation/virt-manager-2.0.0::nokit was built with the following:
USE="gnome-keyring gtk policykit -sasl" PYTHON_TARGETS="python3_7"

[alamahant]

Code: Select all

=================================================================
                        Package Settings
=================================================================

app-emulation/qemu-5.0.0::gentoo was built with the following:
USE="aio alsa bzip2 caps curl fdt filecaps gtk jpeg ncurses nls opengl oss pin-upstream-blobs png pulseaudio sdl seccomp spice usb usbredir vhost-net vnc xattr xkb -accessibility -capstone -debug -doc -glusterfs -gnutls -infiniband -io-uring -iscsi -jemalloc -lzo -nfs -numa -plugins -python -rbd -sasl -sdl-image (-selinux) -smartcard -snappy -ssh -static -static-user -systemtap -tci -test -vde -vhost-user-fs -virgl -virtfs -vte -xen -xfs -zstd" ABI_X86="(64)" PYTHON_TARGETS="python3_7 -python3_6 -python3_8" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -cris -hppa -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -riscv32 -riscv64 -rx -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-aarch64 -aarch64_be -alpha -arm -armeb -cris -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -x86_64 -xtensa -xtensaeb"
FEATURES="binpkg-docompress unmerge-logs news network-sandbox qa-unresolved-soname-deps sfperms parallel-fetch usersandbox ebuild-locks assume-digests distlocks binpkg-logs merge-sync config-protect-if-modified protect-owned strict userpriv preserve-libs sandbox multilib-strict unknown-features-warn usersync unmerge-orphans ipc-sandbox userfetch pid-sandbox binpkg-dostrip fixlafiles xattr"

>>> Attempting to run pkg_info() for 'app-emulation/qemu-5.0.0'
Using:
  app-emulation/spice-protocol-0.14.1
  sys-firmware/edk2-ovmf-201905
    USE=binary
  sys-firmware/ipxe-1.0.0_p20190728
  sys-firmware/seabios-1.12.0
    USE=binary
  sys-firmware/sgabios-0.1_pre8-r1

app-emulation/libvirt-6.2.0-r2::gentoo was built with the following:
USE="caps dbus libvirtd macvtap nls policykit qemu udev vepa virt-network -apparmor -audit -dtrace -firewalld -fuse -glusterfs -iscsi -iscsi-direct -libssh -lvm -lxc -nfs -numa (-openvz) -parted -pcap -rbd -sasl (-selinux) -virtualbox -wireshark-plugins -xen -zfs" ABI_X86="(64)"


app-emulation/virt-manager-2.2.1-r3::gentoo was built with the following:
USE="gtk policykit -sasl" ABI_X86="(64)" PYTHON_SINGLE_TARGET="python3_7 -python3_6 -python3_8"


sys-firmware/edk2-ovmf-201905::gentoo was built with the following:
USE="binary" ABI_X86="(64)"

So it would be best if you installed latest qemu using ~amd64 in /etc/portage/package.accept_keywords

[Moldmaker]

So it would be best if you installed latest qemu using ~amd64 in /etc/portage/package.accept_keywords

I did. I updated to the following versions: app-emulation/qemu-5.0.0 and app-emulation/libvirt-6.2.0-r2 (including the required dependencies), and *now* I can choose the UEFI boot. However, at the beginning of the installation the passthrough of the PCI card fails with this error message:

Code: Select all

Installation konnte nicht fertiggestellt werden: «Interner Fehler: qemu unexpectedly closed the monitor: 2020-06-13T11:13:37.600107Z qemu-system-x86_64: -device vfio-pci,host=0000:05:01.0,id=hostdev0,bus=pci.0,addr=0x6: vfio 0000:05:01.0: failed to setup container for group 9: Failed to set iommu for container: Operation not permitted»

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 2119, in _do_async_install
    guest.installer_instance.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/installer.py", line 419, in start_install
    doboot, transient)
  File "/usr/share/virt-manager/virtinst/installer.py", line 362, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.7/site-packages/libvirt.py", line 4035, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: Interner Fehler: qemu unexpectedly closed the monitor: 2020-06-13T11:13:37.600107Z qemu-system-x86_64: -device vfio-pci,host=0000:05:01.0,id=hostdev0,bus=pci.0,addr=0x6: vfio 0000:05:01.0: failed to setup container for group 9: Failed to set iommu for container: Operation not permitted

What can I do next?

[Moldmaker]

I found something. When I do

Code: Select all

# dmesg | grep -e vfio

the answer is

Code: Select all

[   12.006441] vfio_pci: add [1244:0e00[ffffffff:ffffffff]] class 0x000000/00000000
[  348.959110] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[  770.096133] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

Where do I have to add "allow_unsafe_interrupts"?

[Anon-E-moose]

Code: Select all

$ grep -r interrupt /etc/modprobe.d/
/etc/modprobe.d/options.conf:options vfio_iommu_type1 allow_unsafe_interrupts=1 

[Moldmaker]

Code: Select all

/etc/modprobe.d/options.conf:options vfio_iommu_type1 allow_unsafe_interrupts=1 

Thank you, that worked! Now I can assign the PCI card to the VM without error message. But ... the VM does not boot from the specified iso-image nor from the specified DVD drive. It drops me into an interactive UEFI shell. I can type exit and find some options, so I switched the boot order to the QEMU DVD drive. With no effect, I am stuck in the UEFI shell. I have two options for UEFI,

Code: Select all

/usr/share/qemu/edk2-x86_64-code.fd

and

Code: Select all

/usr/share/edk2-ovmf/OVMF_CODE.fd

Both do the same.

Edit:
I also tried the other chipset, Q35, instead of i440FX, and then I get

Code: Select all

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 2119, in _do_async_install
    guest.installer_instance.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/installer.py", line 419, in start_install
    doboot, transient)
  File "/usr/share/virt-manager/virtinst/installer.py", line 362, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.7/site-packages/libvirt.py", line 4035, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: Nicht unterstützte Konfiguration: IDE controllers are unsupported for this QEMU binary or machine type

Edit 2:
Found something again. When I use another iso, or another DVD, the VM boots into that. So this problem seems to be on my end.
Thank you everyone for guiding me through this. I really appreciate it.