virt-manager: NIC passthrough

[taskman]

Hi,

I am looking for advice to set up "libvirt" and "qemu" with virt-manager.
I can run VMs without network access.
But my goal is to have every VM with its own NIC-passthrough (up to four).
Therefor I have bought a PCIe-CNA/NIC with Intel 82580 chipset.

[NeddySeagoon]

taskman,

Passthrough and bridging are different things.

When you use passthrough, the host does not see the hardware item, its passed through to the VM. The VM 'owns' it.

When you use a bridge, the host may or may not have a connection to the bridge.
The host donates the actual interface to the bridge and if the host will have a connection to the bridge, the bridge gets an IP address, not the interface donated to the bridge.

I have a Intel Corporation 82575GB 4 port card that I wanted to passthrough but it has a hardware bug, so I have to use bridging instead.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[taskman]

Thanks for the clarification.
I don't know what lead me to bridging in the first place.

I added "virt-network" to app-emulation/libvirt and then I needed to compile some additional tools I dont need, like dnsmasq radvd and a few others.
But still I don't know how to create a NIC passthrough.

This is what I have done now ...
Edit > Connection Details > Add Network > Forwarding to physical network > Mode: Routed

Then I need to choose a device and this lead in one of the following errors ...

w/o IPv4 network address space definition ...

[NeddySeagoon]

taskman,

There are four things you can do.

a) libvirt will set up NAT by default, using 192.168.122.0/24 (No IPv6). This is free with USE=virt-network
b) Donate an interface to a bridge and connect the KVM(s) to a bridge. The host may or may not connect to this bridge.
c) Create an empty bridge, no real hardware at all, then route traffic to it. The guests can use this bridge.
d) Pass the NIC to the KVM. The host runs a PCI-Stub driver and the Guest sees the real hardware.

I do all of the first three. I would prefer d) but my hardware has a bug, so it won't work for me.

You appear not to want a)
Which of the other solutions do you want?
I can't help much with d)
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[taskman]

My goal is to have up to four VMs with its own NIC.
- switching between Windows and some amd64- and ARM-Distributions (w/ one NIC)
- PFSense (w/ two NICs)
- Kali for hacking/pentesting other VMs (w/ one NIC)

Most things are for learning purpose, therefor I want Option D I think.
Maybe bridging will work too, but this will result in a performance lost and that I dont want, cause my next step will be Windows-Gaming in a VM w/ GPU-passthrough.

[NeddySeagoon]

taskman,

Here's an outline of bridging. This is on the host. My firewall/router runs as a KVM on this host.
udev is not permitted to renawe the interfaces.

[Anon-E-moose]

This might have some clues (follow the links to ubuntu documentation) https://askubuntu.com/questions/1028489/intel-pci-e-quad-port-card-passthrough

I pass through a pcie nic card but it's a single port card and likely to be far different than what you're trying to do.

Edit to add: I just looked at the first post since they have individual IP addresses, you might be able to use regular pass through.

Basically unbind them from the linux side and bind them selectively to the virtual

# unbind addon ethernet from linux
echo 0000:03:00.0 > /sys/bus/pci/devices/0000:03:00.0/driver/unbind

set up virtual side
modprobe vfio_pci (if not loaded)
# ethernet card (from lspci -nnk or similar)
echo "10ec 8168" >/sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:03:00.0 >/sys/bus/pci/drivers/vfio-pci/bind

then let virtual know about it
qemu-system-x86_64 --enable-kvm -m 1024 -cpu athlon -vga std -net none -device vfio-pci,host=03:00.0 /n/don/virtual/XP.img

#unbind devices when done

Of course all the id's and addresses, etc have to be changed for what your system has.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[taskman]

Thanks all for helping me and happy Xmas.

I've using NIC-passthrough now, but still a bit of a mess with unbind.

This is what I have done so far ...

I added iommu=pt iommu=1 to my grub.cfg.
FYI: For Intel CPUs you should add intel_iommu=on instead.

[Anon-E-moose]

I'm not sure why but I don't have a multi-ethernet card to check with.

What does "ls /sys/bus/pci/devices/0000\:02\:00.*" return? Is there a driver subdir under each of them, and does it have an unbind under it?

And what does "ls /sys/bus/pci/drivers" return?
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[taskman]

[Anon-E-moose]

I'm not sure why it's grabbing 2-4 eth but freeing them involves something like

echo 0000:02:00.2 > /sys/bus/pci/drivers/igb/unbind
then
echo 0000:02:00.2 > /sys/bus/pci/devices/0000:02:00.2/driver/unbind

I would just do the one and make sure it frees from the driver then gets unbound, then you should be able to do the others.

It sounds like some of the eth "cards" are being grabbed before the original attempt at an unbind on all of them works, may be a timing issue.

I put all my unbind of drivers/devices in /etc/local.d/baselayout1.start but that might not be far enough in the boot process for you to get everything done.
You could put all the unbind drivers/devices in a separate script and have it execute after you're sure all the booting has been done.

Edit to add: The way I do it (inside a shell script as I bring up a vm) is

[gjy0724]

It sounds like you are looking for something similar to what I am doing. I have two physical interfaces on my desktop, one dedicated for my desktop, the second dedicated of use by my VMs within KVM/qemu/libvirt. I am using systemd so if you are using openrc then this will likely not work. I create a bridge (br0) then tie my second virtual interface to the bridge. Then I set all VMs to use the bridge (br0) interface. Then the VM has an IP or IPs, if there are multiple interfaces, on the local network so it can reach the internet and is accessible locally as well.

My post on this issue can be found here: https://forums.gentoo.org/viewtopic-t-1063212-highlight-gjy0724.html

The end result was the following, which I am currently using:

/etc/systemd/network/br0.netdev

[taskman]

Soon I will buy a new computer and there I will add systemd to get rid of log files.
But for now I run with initrd and this logs makes me nuts.

ATM I am banging my head on the keyboard (since 12 hours) to get a DIY Kernel.
Then I try dracut to set up rd.driver.pre hook with vfio.

I wasn't able to put vfio into initramfs so far.