Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Network configuration for KVM with nftables
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Fri Apr 15, 2022 2:04 pm    Post subject: [SOLVED] Network configuration for KVM with nftables Reply with quote

I need help manually configuring the network for my QEMU/KVM guests. I would like to have two groups: VMs that can connect to the internet and VMs that are isolated, each with their own nftables rules.
I use libvirtd and virt-manager GUI to manage my virtual machines. It seems like the network configuration does not work out of the box because I use nftables instead of iptables.
Also I don't use firewalld or systemd, like most of the online tutorials do that I found. Actually I prefer setting up the network manually and learning along the way.

I imagine something like this:
Code:

         ┌────────────────────┐
         │ Internet           │
         │                    │
         └──────────┬─────────┘
                  ▲ │
                  │ │
                  │ ▼
         ┌────────┴───────────┐
         │ Router             │
         │ DNS server         │
         │ DHCP server        │
         │                    │
         │ 192.168.1.1        │
         └──────────┬─────────┘
                  ▲ │
                  │ │ Local subnet
                  │ │ 192.168.1.0/24
                  │ │
                  │ ▼
                 ┌┼─┬┐ wlp0s20f3
┌────────────────┼┼┼┼┼──────────────────────────────────────────────┐
│                └┴─┼┘                                  Gentoo host │
│                 ▲ │                                 192.168.1.100 │
│              ───┼─┼─── Existing nftables firewall                 │
│  ┌──────────────┼─┼──────────────────────────────────────┐        │
│  │              │ │                                      │        │
│  │              │ ▼       ┌──────────────────────────┐   │        │
│  │             ┌┴──┐  │   │ QEMU/KVM Guest           │   │        │
│  │      virbr0 │   ├──┼──►│                          │   │        │
│  │ 192.168.2.1 │   │  │   │                          │   │        │
│  │             │   │◄─┼───┤                          │   │        │
│  │             └───┘  │   │                          │   │        │
│  │                    │   └──────────────────────────┘   │        │
│  │                     nftables                          │        │
│  │                     firewall                          │        │
│  │ Subnet for VMs                                        │        │
│  │ 192.168.2.0/24                                        │        │
│  └───────────────────────────────────────────────────────┘        │
│                                                                   │
│  ┌───────────────────────────────────────────────────────┐        │
│  │                       virbr1                          │        │
│  │  ┌───────────────┐    ┌────┐    ┌─────────────────┐   │        │
│  │  │ QEMU/KVM      ├───►│    ├───►│ QEMU/KVM        │   │        │
│  │  │ Guest 1       │    │    │    │ Guest 2         │   │        │
│  │  │               │◄───┤    │◄───┤                 │   │        │
│  │  └───────────────┘    └────┘    └─────────────────┘   │        │
│  │ Isolated subnet, 192.168.3.0/24                       │        │
│  └───────────────────────────────────────────────────────┘        │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘


libvirtd created the "default" virbr0 bridge on my system. For the guests, I selected this bridge in "NAT" mode in virt-manager.
I am not sure if this is correct or if I have to select something else. I am also confused how I need to configure NAT with nftables.
I tried
Code:

table inet nat {
chain nat {
      type nat hook postrouting priority 100; policy accept;
      ip saddr 192.168.2.0/24 counter snat to 192.168.1.100;
   }
}

but that does not work (guest can not access internet); I think I have not entirely understood how NAT works. As far as I understand,
it allows to translate between two subnets. But do I have to specify a special port for this (on my host)?
I can see in wireshark that my guests communicates with the bridge (which confuses me, because in my understanding, bridging happens only in layer 2...)

I would very much appreciate if someone could share their configuration or give me some ideas on how I could set this up. I looked around in the QEMU, KVM and Nftables wikis, but I couldn't find anything that would help me.

Thanks in advance.


Last edited by regox on Sun Apr 17, 2022 4:02 pm; edited 2 times in total
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Fri Apr 15, 2022 3:45 pm    Post subject: Re: Network configuration for KVM with nftables Reply with quote

regox wrote:
It seems like the network configuration does not work out of the box because I use nftables instead of iptables.
As far as I know libvirt specifically prefers to use libnft via firewalld. Make sure you have enabled firewalld USE flag for app-emulation/libvirt and nftables use flag for net-firewall/firewalld. You may also need to enable nftables use flag for net-misc/networkmanager too. And of course have kernel compiled for nftables support (which I assume you have already).

Lastly, remember to set
Code:
net.ipv4.ip_forward = 1
in sysctl.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Fri Apr 15, 2022 4:21 pm    Post subject: Re: Network configuration for KVM with nftables Reply with quote

Thank you for your reply.

Zucca wrote:
regox wrote:
It seems like the network configuration does not work out of the box because I use nftables instead of iptables.
As far as I know libvirt specifically prefers to use libnft via firewalld. Make sure you have enabled firewalld USE flag for app-emulation/libvirt and nftables use flag for net-firewall/firewalld. You may also need to enable nftables use flag for net-misc/networkmanager too. And of course have kernel compiled for nftables support (which I assume you have already).


Okay, I understand that firewalld is needed so that libvirtd can manage the network "itself". But is it not possible to do this without firewalld, with a manual configuration?
I have my (host) nftables rules set up without it and I am happy the way it currently is, I am concerned that firewalld may interfere with the current setup. I am also not happy with the fact that firewalld would create a kind of redundancy, since I already use nft to set my rules, I don't need another management tool just for VMs; or am I mistaken what firewalld actually does?

Zucca wrote:

Lastly, remember to set
Code:
net.ipv4.ip_forward = 1
in sysctl.

Sorry I didn't mention. It is set correctly.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Fri Apr 15, 2022 7:27 pm    Post subject: Reply with quote

Firewalld may indeed interfere with your custom configs. I know there was an ability to insert "raw" iptables rules before. At that time nftables "raw" rules were not a thing. I don't know the current state of things. This whole virt-manager, firewalld and networkmanager combination is kind of a RedHat walled garden. :( I'm actually quite fine with firewalld, but not with networkmanager.
If I had my server running I'd give you my nft configs.

I'll try to look for them in another place.

EDIT: Maybe see this post for NAT.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Sun Apr 17, 2022 12:45 pm    Post subject: Reply with quote

Okay, I have made some progress. I decided to focus on the bridge with internet access ("virbr0" in my previous post).

I use this script to set up the bridge:
Code:

BR_NAME_VM=vm-bridge

# Create bridge device
ip link add name $BR_NAME_VM type bridge

# Set state up
ip link set $BR_NAME_VM up

# Associate IP address to bridge device
ip addr add 192.168.2.1/24 brd + dev $BR_NAME_VM

# Delete the default route
ip route delete 192.168.2.0/24

# Add route
ip route add 192.168.2.0/24 dev $BR_NAME_VM proto dhcp

This leads to this:
Code:

16: vm-bridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 1e:b9:a5:35:78:81 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global vm-bridge
       valid_lft forever preferred_lft forever

and these routes:
Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.236.229 0.0.0.0         UG    600    0        0 wlp0s20f3
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 vm-bridge
192.168.236.0   0.0.0.0         255.255.255.0   U     600    0        0 wlp0s20f3

Please excuse me that the local IPs do not match my initial post. I currently use my Android phone as an access point for my laptop, so I can't change the IPs.
Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host):
Code:

interface=vm-bridge
bind-dynamic

no-poll
no-resolv

dhcp-range=192.168.2.2,192.168.2.255

server=127.0.0.1#8934

On my hosts 127.0.0.1:8934 there is an upstream DNS resolver listening.

My hosts nftables config is this:
Code:

table ip nat {
   chain prerouting {
      type nat hook prerouting priority 0; policy accept;
      log prefix "prerouting: " flags all
   }

   chain postrouting {
      type nat hook postrouting priority 100; policy accept;
      oifname "vm-bridge" masquerade
      log prefix "postrouting: " flags all
   }
}
table inet filter { # handle 6
   chain input { # handle 1
      type filter hook input priority filter; policy drop;
      ct state { established, related } accept # handle 5
      ct state invalid drop # handle 6
      iifname "lo" accept # handle 7
      iifname "vm-bridge" accept
      ...
   }

   chain forward { # handle 2
      type filter hook forward priority filter; policy drop;
                ...
      iifname "vm-bridge" accept
      oifname "vm-bridge" accept
      ...
   }

   chain output { # handle 3
      type filter hook output priority filter; policy drop;
      oifname "vm-bridge" accept
                ...
   }
}


With this, my Ubuntu QEMU/KVM guest can successfully get an IP, Gateway and DNS information over DHCP:
Code:

IPv4:               192.168.2.225
Gateway:            192.168.2.1
DNS:                192.168.2.1

DNS works from within the guest (verified with manual queries using dig). From the host, I can also ping the guest and vice versa. So far so good.

The problem is that the guest can not connect to the internet (ping, curl or Firefox). If I let the guest continuously ping a public IP, I can check with wireshark that those packets are on vm-bridge, but they never leave my physical interface, wlp0s20f3. In dmesg, there is this:
Code:

Apr 17 14:16:32 thinkpad kernel: prerouting: IN=vm-bridge OUT= MACSRC=52:54:00:81:13:bf MACDST=1e:b9:a5:35:78:81 MACPROTO=0800 SRC=192.168.2.225 DST=216.58.212.174 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9948 DF PROTO=ICMP TYPE=8 CODE=0 ID=10 SEQ=1105

but no postrouting (?)

So there is still a problem with NAT and/or the routing table. Can anyone help?
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Sun Apr 17, 2022 4:02 pm    Post subject: Reply with quote

I think it should be like this:
Code:

table ip nat {
      chain postrouting {
         type nat hook postrouting priority 100;
         policy accept;
         ip saddr 192.168.2.0/24 masquerade
   }
}


That seems to work for now.
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Mon Sep 12, 2022 6:49 pm    Post subject: Reply with quote

I'm trying to manually setup nftables for qemu just like you

About the

> On my hosts 127.0.0.1:8934 there is an upstream DNS resolver listening

do you need to set this beforehand? I'm just using NetworkManager, and never tweaked anything manually.

Also, I'm wondering: are all of the steps (creating bridge interfaces, IPs, etc) "safe"? I'm concerned with messing up my setup beyond my fixing capabilities (I don't really know much about any of this) :oops:

Cheers.
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Tue Sep 13, 2022 10:44 am    Post subject: Reply with quote

mani001 wrote:

do you need to set this beforehand? I'm just using NetworkManager, and never tweaked anything manually.


Yes. I use stubby for DNS-over-TLS which I set up to listen on localhost:8934. In /etc/NetworkManager/conf.d/20-dnsmasq.conf I have disabled NetworkManager's dnsmasq with
Code:

[main]
dns=none

Instead I use a standalone instance of dnsmasq which gets started automatically by OpenRC (default runlevel). This instance is listening on the normal DNS port 53 and using localhost:8934 as an upstream server.

If you don't want or need DNS-over-TLS, you could change localhost:8934 to localhost:53 in the dnsmasq configuration for the virtual machines. This way, this instance will use your host's dnsmasq instance as an upstream server (which should have been started by NetworkManager by default I think).

mani001 wrote:

Also, I'm wondering: are all of the steps (creating bridge interfaces, IPs, etc) "safe"? I'm concerned with messing up my setup beyond my fixing capabilities (I don't really know much about any of this)


I think it's pretty safe because creating bridges and assigning IPs is not permanent. After a reboot, your configuration will be lost. I re-run my script every time I start my KVMs. I would make backups of dnsmasq's and NetworkManager's configuration files, though.

Let me know if this works.

Cheers, regox
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sat Sep 17, 2022 3:30 pm    Post subject: Reply with quote

Thanks for your help.

A couple of quick questions before stirring up things:

- do you make a new interface, vm-bridge and just ignore the virbr0 that is (automatically) created by libvirtd?

Quote:
Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host)


How do you start another instance? Manually (if so, how?)? Openrc? Right now, I'm using Networkmanager to start the one used by the host.

Cheers.
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Mon Sep 19, 2022 11:46 am    Post subject: Reply with quote

mani001 wrote:

- do you make a new interface, vm-bridge and just ignore the virbr0 that is (automatically) created by libvirtd?

Yes.

mani001 wrote:

Quote:
Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host)


How do you start another instance? Manually (if so, how?)? Openrc? Right now, I'm using Networkmanager to start the one used by the host.

Manually with
Code:

sudo dnsmasq -C vm-bridge.conf

where vm-bridge.conf contains the configuration I posted beforehand.

Cheers, regox
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sat Sep 24, 2022 9:00 am    Post subject: Reply with quote

Thanks for all the tips, but I'm still struggling with this :oops:

In particular, when I try to run another instance of dnsmasq for the new interface, I go from

Code:
root@cochi ~ $ss -alpn src 127.0.0.1
Netid State  Recv-Q Send-Q  Local Address:Port   Peer Address:Port Process                             
udp   UNCONN 0      0           127.0.0.1:53          0.0.0.0:*     users:(("dnsmasq",pid=370,fd=4))   
tcp   LISTEN 0      0           127.0.0.1:8384        0.0.0.0:*     users:(("syncthing",pid=427,fd=16))
tcp   LISTEN 0      0           127.0.0.1:631         0.0.0.0:*     users:(("cupsd",pid=296,fd=8))     
tcp   LISTEN 0      0           127.0.0.1:53          0.0.0.0:*     users:(("dnsmasq",pid=370,fd=5))   


to

Code:
root@cochi ~ $ss -alpn src 127.0.0.1                                                                   
Netid State  Recv-Q Send-Q  Local Address:Port   Peer Address:Port Process                             
udp   UNCONN 0      0           127.0.0.1:53          0.0.0.0:*     users:(("dnsmasq",pid=3757,fd=8))   
udp   UNCONN 0      0           127.0.0.1:53          0.0.0.0:*     users:(("dnsmasq",pid=370,fd=4))   
tcp   LISTEN 0      0           127.0.0.1:8384        0.0.0.0:*     users:(("syncthing",pid=427,fd=16))
tcp   LISTEN 0      0           127.0.0.1:631         0.0.0.0:*     users:(("cupsd",pid=296,fd=8))     
tcp   LISTEN 0      0           127.0.0.1:53          0.0.0.0:*     users:(("dnsmasq",pid=370,fd=5))


i.e., a new instance of dnsmasq is listening (but only for UDP) at port 53 8O

Looking at this post https://unix.stackexchange.com/questions/690523/dnsmasq-failed-to-create-listening-socket-for-127-0-0-1-address-already-in-use I thought it was because the systemd-started instance of dnsmasq already handles every interface. I tried setting

Code:
except-interface=vm-bridge


in /etc/dnsmasq.conf, but it didn't help (same result). Any clue?
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Sun Sep 25, 2022 9:14 am    Post subject: Reply with quote

mani001 wrote:
in /etc/dnsmasq.conf

Are you sure NetworkManager uses this file? I think /etc/dnsmasq.conf is just the default config if you start dnsmasq manually or via OpenRC.

Instead, I would try to change /etc/NetworkManager/dnsmasq.q/dnsmasq.conf.

Maybe you also want to open up a new thread so more people can see it (and link it here).
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sun Sep 25, 2022 12:22 pm    Post subject: Reply with quote

Hi,

OK, I think I got confused with all the dnsmasq stuff. Since I don't need stubby, (as you said) I can just setup localhost:53 in the dnsmasq configuration for the virtual machines.

Now, when I use your nft configuration, my internet connection is blocked because of the *chain output* part. I guess it was to be expected since the policy is "drop", and when I comment that out, it's fine. I take it, you fix that in the "...". Anyway, even If switch the policy to accept, DNS is not working from within the VM (there is internet connection, though, since I can ping google or bing).

In virt-manager, you just choose "Bridge device" as "Network source" and then fill in "vm-bridge" as the "Device name", right? Right now I have

Code:
<interface type="bridge">
  <mac address="52:54:00:5d:d5:75"/>
  <source bridge="vm-bridge"/>
  <target dev="vnet1"/>
  <model type="e1000e"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>



Cheers.
Back to top
View user's profile Send private message
regox
Tux's lil' helper
Tux's lil' helper


Joined: 12 Sep 2021
Posts: 79

PostPosted: Sun Sep 25, 2022 2:07 pm    Post subject: Reply with quote

mani001 wrote:

Now, when I use your nft configuration, my internet connection is blocked because of the *chain output* part. I guess it was to be expected since the policy is "drop", and when I comment that out, it's fine. I take it, you fix that in the "...".

Correct. In the "..." I specify all ports I need.

mani001 wrote:

Anyway, even If switch the policy to accept, DNS is not working from within the VM (there is internet connection, though, since I can ping google or bing).

Is DHCP working? Maybe you can try to manually set a static IP, gateway and DNS inside the VM.


mani001 wrote:

In virt-manager, you just choose "Bridge device" as "Network source" and then fill in "vm-bridge" as the "Device name", right? Right now I have

Code:
<interface type="bridge">
  <mac address="52:54:00:5d:d5:75"/>
  <source bridge="vm-bridge"/>
  <target dev="vnet1"/>
  <model type="e1000e"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>


Yes.

By the way, some helpful tools to debug:
- net-analyzer/wireshark: you can set it up to listen on vm-bridge, so you exactly know which packets are being sent
- dig command: Test your dnsmasq servers, with dig @127.0.0.1 -p 53 gentoo.org (test both host and within VM)
- you can use additional "log" statements in the nftables chains so packets get logged to dmesg

Cheers, regox
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sun Sep 25, 2022 8:18 pm    Post subject: Reply with quote

Using dig before starting libvirtd and booting the VM I get

Code:
manu@cochi ~ dig @127.0.0.1 -p 53 gentoo.org

; <<>> DiG 9.16.30 <<>> @127.0.0.1 -p 53 gentoo.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39366
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;gentoo.org.                    IN      A

;; ANSWER SECTION:
gentoo.org.             473     IN      A       151.101.2.137
gentoo.org.             473     IN      A       151.101.66.137
gentoo.org.             473     IN      A       151.101.130.137
gentoo.org.             473     IN      A       151.101.194.137

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 25 22:02:05 CEST 2022
;; MSG SIZE  rcvd: 103


which looks good. However, after starting libvirtd and booting the VM I get

Code:
manu@cochi ~ dig @127.0.0.1 -p 53 gentoo.org

; <<>> DiG 9.16.30 <<>> @127.0.0.1 -p 53 gentoo.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 48282
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; EDE: 14 (Not Ready)
;; QUESTION SECTION:
;gentoo.org.                    IN      A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 25 22:06:54 CEST 2022
;; MSG SIZE  rcvd: 45


i.e., the server refuses to answer, and I can see tons of related messages in Wireshark, e.g.,

Code:
728   350.895281544   192.168.2.203   192.168.2.1   DNS   70   Standard query 0x826a A g.live.com
729   350.895343193   192.168.2.1   192.168.2.203   DNS   70   Standard query response 0x826a Refused A g.live.com


Could it be due to the iptables rules that libvirt adds (I'm hunting high and low here...)

Code:
root@cochi qemu_nftables $iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_FWX  all  --  0.0.0.0/0            0.0.0.0/0           
LIBVIRT_FWI  all  --  0.0.0.0/0            0.0.0.0/0           
LIBVIRT_FWO  all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:68



PD: dig and wireshark are pretty cool tools!!
Back to top
View user's profile Send private message
pietinger
Moderator
Moderator


Joined: 17 Oct 2006
Posts: 4008
Location: Bavaria

PostPosted: Sun Sep 25, 2022 8:58 pm    Post subject: Reply with quote

mani001 wrote:
Could it be due to the iptables rules that libvirt adds (I'm hunting high and low here...)

With "iptables -L -n" we dont see your device. We would need "iptables -L -v -n"
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sun Sep 25, 2022 9:54 pm    Post subject: Reply with quote

Good point...It seems that libvirtd is only tweaking stuff for virbr0 (which should be fine, I guess)

Code:
root@cochi qemu_nftables $iptables -L -nv
Chain INPUT (policy ACCEPT 205K packets, 92M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 205K   92M LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 2 packets, 128 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   128 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    2   128 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    2   128 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 278K packets, 226M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 278K  226M LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_FWI (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_INP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:68
Back to top
View user's profile Send private message
pietinger
Moderator
Moderator


Joined: 17 Oct 2006
Posts: 4008
Location: Bavaria

PostPosted: Sun Sep 25, 2022 10:15 pm    Post subject: Reply with quote

mani001 wrote:
Good point...It seems that libvirtd is only tweaking stuff for virbr0 (which should be fine, I guess)

Yes ... BUT ... Only to be safe I have to tell you that your host is not protected by your firewall, because default policy/action for INPUT and OUTPUT is ACCEPT (so, both Chains "LIBVIRT_INP" and "LIBVIRT_OUT" are senseless, because after these accepts all other traffic is allowed also).
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Mon Sep 26, 2022 7:09 am    Post subject: Reply with quote

I'm not actually using iptables, and my rules are in nftables. As far as I know both firewalls act on the system so I'm fine as long as I have set up rules in nftables...right? :roll: When I switched from iptables to nftables (about 1 year ago), I couldn't find any way to "turn off" iptables (I guess I should disable it in the kernel...maybe?). I just posted the iptables rules because libvirtd make changes there which I don't really understand (and have no control over).
Back to top
View user's profile Send private message
pietinger
Moderator
Moderator


Joined: 17 Oct 2006
Posts: 4008
Location: Bavaria

PostPosted: Mon Sep 26, 2022 10:12 am    Post subject: Reply with quote

mani001 wrote:
I just posted the iptables rules because libvirtd make changes there which I don't really understand (and have no control over).

I dont know libvirt but I cant believe that it is not able to support nftables also. (The rules from libvirt are ok).
If you want disable iptables you can remove its start at boot-time (OpenRC: It is /etc/init.d/iptables). Which doesnt help if some systems sets up some ipatbles commands - yes, in this case you must remove kernel modules. But a better way would be to tell libvirt doing nftables commands.
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Mon Sep 26, 2022 11:28 am    Post subject: Reply with quote

Quote:
I dont know libvirt but I cant believe that it is not able to support nftables also. (The rules from libvirt are ok).


I know...it's kind of shocking since I wouldn't say nftables is nowadays "cutting edge" :-/ One is proably expected to let firewalld manage everything as suggested above. Also, my first attemp was to let libvirtd deal with it using "xtables-nft-multi" iptables implementation (eselect iptables...), which I though was some kind of adapter. It didn't work.

Quote:
If you want disable iptables you can remove its start at boot-time (OpenRC: It is /etc/init.d/iptables). Which doesnt help if some systems sets up some ipatbles commands - yes, in this case you must remove kernel modules. But a better way would be to tell libvirt doing nftables commands.


I don't know anymore about openrc, but I think the only thing the equivalent systemd service does is to restore the saved rules (meaning, if you disable it, you just have iptables working with the default, i.e., none, rules). I guess dropping support in the kernel must indeed be the way.
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Mon Sep 26, 2022 7:00 pm    Post subject: Reply with quote

An update...I was wrong about the REFUSED errors in dig: they have nothing to do with libvirtd but start to happen when I start another instance of dnsmasq with the above setup replacing the port, i.e.,

Code:
interface=vm-bridge
bind-dynamic

no-poll
no-resolv

dhcp-range=192.168.2.2,192.168.2.255

server=127.0.0.1#53


I guess dig is asking this new instance of dnsmasq, which only listens to interface vm-bridge. I think my problem is I don't get how dnsmasq works... For instance, shouldn't I tell dnsmasq to listen in a different port (e.g., port=54) so that I don't have Networkmanager's and my own instance both listening on port 53 (or setup Networkmanager to start its instance at some other port)?
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Sat Oct 01, 2022 8:09 am    Post subject: Reply with quote

I simplified the dnsmasq thing and now I just start a single dnsmasq instance (through systemd) with the following parameters in /etc/dnsmasq.conf:

Code:
port=5028 # standard port 5353 was already in use
bind-dynamic
interface=vm-bridge
dhcp-range=192.168.2.2,192.168.2.255


(I don't know whether it is to be expected but this dnsmasq server, despite the interface=vm-bridge, still answers requests that I write (dig) in any terminal)

The result is apparently the same as before: the VM has internet connection (I can ping sites) but no DNS server. One thing I don't get is: if the VM got hold of an IP in the above range (served by dnsmasq), that must mean that it has access to the dnsmasq instance (right?)...but then, why is not using it also for DNS?

In the Windows VM, I checked the network properties, and I found indeed that the DNS server is missing. I took a screenshot (it's in Spanish, sorry): https://ibb.co/vDnS3VZ

Any other clue?
Back to top
View user's profile Send private message
mani001
Guru
Guru


Joined: 04 Dec 2004
Posts: 481
Location: Oleiros

PostPosted: Wed Oct 12, 2022 11:17 am    Post subject: Reply with quote

I finally solved this by choosing and adapting the above rules. I posted the final solution in this thread

https://forums.gentoo.org/viewtopic-p-8674890.html#8674890

Thanks for all the help!!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum