[SOLVED] IOMMU groups - change/assign/rearrange

[manwe_]

Hi *.

I have i7-4790K with ASUS Z97-AR. This motherboard is equipped with two PCI-E 3.0 ×16 slots. Few days ago I've changed my Samsung SSD 850 (SATA) to M.2 960 Evo. But as it turned out, on this motherboard M.2 slot is only PCI-E 2.0 ×4 (10Gbps). Therefore my disk was limited to ~900MB/s. I bougth M.2 to full PCI-E adapter and inserted the disk into second PCI-E 3.0 slot. And it worked great, reaching around 1.7GB/s on read and 1.3GB/s on write.

But. There always has to be "but". Unfortunately both PCI-E 3.0 ×16 slots share IOMMU group. And in the first slot I have GTX 960 for PCI passthrough to Windows 10 (qemu/kvm). With disk in the same group I was unable to start qemu.

For now I've moved the disk into third "large" slot, but this one is only PCI-E 2.0, so I'm back to ~900MB/s. Any ideas? Is there a way for me to have both GPU-passthrough and PCI-E 3.0 on SSD?

[tholin]

I'm not familiar with your motherboard but Z97 boards are basically all the same. They have two pcie buses. One going directly to the cpu with 3.0 speed and another on the chipset with 2.0 speed. The i7-4790K cpu doesn't have support for ACS so all devices in 3.0 slots end up in the same group. The chipset slots are separated but with lower speed. The only way to work around the problem is to use the out of tree ACS override patch. It makes it possible to forcefully override iommu groups. But iommu groups exist for a reason and overriding them can result in system instability. The only way to know if it works for you is to test it.

[manwe_]

I've added the patch and pcie_acs_override=downstream to cmdline and it worked like a charm PCI-E 01:00.0 (VGA) is now in different group then PCI-E 02:00.0 (SSD) and qemu starts without complaining. Thank you!