Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
SOLVED: NF Tables Router + VM
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
rickvernam
Guru
Guru


Joined: 09 Jul 2004
Posts: 310

PostPosted: Thu Jun 04, 2015 7:45 pm    Post subject: SOLVED: NF Tables Router + VM Reply with quote

I have a setup that I've used successfully with kernel 3.13 through 3.17 on a router within my LAN.

Code:
                               "Router"
                           ________________________________________________________________
                           |                                                              |
                           |   br1 (192.168.1.7)                    (10.2.2.1) br0        |
192.168.1.0/24             |   / --tap0--(192.168.1.24)--VM--(10.2.2.2)--tap1-- \         |
---------------------------|---                                                  ---------|---------- 10.2.2.0/23
                           |   \ --enp11s0                                      /         |
                           |                                                  enp4s0f0    |
                           |                                                  enp3s0f0    |
default gateway            |                                                  enp3s0f1    |
---------------------------|--- enp4s0f1 (wan ip address)                                 |
                           |--------------------------------------------------------------|


The router is connected to three networks
1 - directly to a modem to an ISP. This is the "Router's" default gateway.
2 - to a 10.2.2.0/23 network. "Router" is the gateway for the 10.2.2.0/23 network. It provides dhcp & some other network services.
3 - to a 192.168.1.0/24 network, which has some other stuff and also it's own gateway through a different ISP. "Router" routes some destinations through this network instead of the default gateway. "Router" also does NAT for stuff going into the 192.168.1.0/24 network.

"Router" runs a QEmu virtual machine for a network appliance which needs to be accessible from both the 192.168.1.0/24 network & the 10.2.2.0/23 network. Therefore, I have two tun/tap interfaces connected to the VM; each tap interface is then bridged to the appropriate interface.
QEmu provides two network interfaces to the VM via:
Code:
-net tap,ifname=tap0,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:01
-net tap,ifname=tap1,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:02


The problem I'm experiencing...
Starting with kernel 3.18, but only when the VM is running, dhcp traffic from the 10.2.2.0/23 network ends up on the 192.168.1.0/24 network, despite nftables rules in "Router" to block that traffic from leaving the 10.2.2.0/23 network - these same rules worked in previous linux versions from 3.13 to 3.17.
I have tcpdump packet captures on the 192.168.1.0/24 dhcp server with dhcp packets containing 10.2.2.0/23 source, whereas other traffic (legitimately) originating in the 10.2.2.0/23 network is correctly NAT'ed to show 192.168.1.7 source address.
I don't believe the guest OS within the VM has anything to do with this; I continue to experience the problem when I run the VM without an HD (ie, no OS running inside the VM).
As you might expect if I remove either tap device from it's bridge, the problem ceases. Once I add the tap device back into it's bridge, it starts back up again.
I am trying to prevent that dhcp traffic from "leaking" onto the 192.168.1.0/24 netowrk; ie, the 10.2.2.0/23 has it's own dhcp server, and dhcp traffic should not traverse between the two networks. Any thoughts as to why this might have started and/or how I may fix the problem?

"Router's" /etc/conf.d/net file is like so (wan ip addresses edited to a.b.c.d, e.f.g.h & i.j.k.l):
Code:
# ===========================================================
# tap0 is used by VM Appliance on the 192.168.1.0/24 network
# ===========================================================

tuntap_tap0="tap"
config_tap0="null"
config_enp11s0="null"

config_br1="192.168.1.7/24"
bridge_br1="enp11s0 tap0"
rc_net_br1_need="net.enp11s0 net.tap0"
mac_br1="52:54:00:12:01:fe"
routes_br1=" 192.168.1.0/24 dev br1
a.b.c.d/29 via 192.168.1.1
e.f.g.h/29 via 192.168.1.1
192.168.2.0/24 via 192.168.1.1
192.168.4.0/24 via 192.168.1.1"


# ===========================================================
# tap1 is used by VM Appliance on the 10.2.2.0/23 network
# ===========================================================

config_enp3s0f0="null"
config_enp3s0f1="null"
config_enp4s0f0="null"

tuntap_tap1="tap"
config_tap1="null"

config_br0="10.2.2.1/23"
bridge_br0="enp3s0f0 enp3s0f1 enp4s0f0 tap1"
rc_net_br0_need="net.enp3s0f0 net.enp3s0f1 net.enp4s0f0 net.tap1"
mac_br0="52:54:00:12:01:ff"


# ===========================================================
# This port will go down the secondary ISP pipe and become the
# "primary" gateway.  It will not bridge with anything.
# ===========================================================

config_enp4s0f1="i.j.k.l netmask 255.255.255.248"
routes_enp4s0f1="default via m.n.o.p"


Bridge info:
Code:
localhost ~ # brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.5254001201ff       no              enp3s0f0
                                                        enp3s0f1
                                                        enp4s0f0
                                                        tap1
br1             8000.5254001201fe       no              enp11s0
                                                        tap0


NF Tables rules:
Initially, starting with 3.13 through 3.17, I did not have the routing nf table & the filter forward chain was just dport 67 & priority 0. The additional rules in the filter forward chain & the routing table were attempts to stop this "leakage."
Code:
localhost ~ # nft list tables
table nat
table filter
table routing


localhost ~ # nft list table nat -nnna
table ip nat {
        chain post {
                 type nat hook postrouting priority 0;
                 ip saddr 10.2.2.0/23 ip daddr a.b.c.d/29 oif br1 snat 192.168.1.7 # handle 2
                 ip saddr 10.2.2.0/23 ip daddr 192.168.1.0/24 oif br1 snat 192.168.1.7 # handle 3
                 ip saddr 10.2.2.0/23 ip daddr 192.168.2.0/24 oif br1 snat 192.168.1.7 # handle 4
                 ip saddr 10.2.2.0/23 ip daddr 192.168.4.0/24 oif br1 snat 192.168.1.7 # handle 5
                 ip saddr 10.2.2.0/23 ip daddr e.f.g.h/29 oif br1 snat 192.168.1.7 # handle 6
                 ip saddr 10.2.2.0/23 oif enp4s0f1 snat i.j.k.l # handle 7
        }

        chain pre {
                 type nat hook prerouting priority 0;
                  (edited out for brevity)
        }
}


localhost ~ # nft list table filter -nnna
table ip filter {
        chain incoming {
                 type filter hook input priority 0;
                 tcp flags & (fin | syn) == fin | syn drop # handle 2
                 tcp flags & (syn | rst) == syn | rst drop # handle 3
                 tcp flags & (fin | syn | rst | psh | ack | urg) < fin drop # handle 4
                 tcp flags & (fin | syn | rst | psh | ack | urg) == fin | psh | urg drop # handle 5
                 ct state { related, established} accept # handle 6
                 ct state invalid drop # handle 7
                 iifname "lo" accept # handle 8
                 ip saddr 10.2.2.0/23 accept # handle 9
                 ip saddr a.b.c.d/29 tcp dport { 2049, 3690, 22, 5432} accept # handle 10
                 ip saddr 192.168.4.0/24 tcp dport { 2049, 22, 1247, 3690, 5432} accept # handle 11
                 ip saddr 192.168.2.0/24 tcp dport { 3690, 5432, 1247, 2049, 22} accept # handle 12
                 ip saddr 192.168.1.0/24 tcp dport { 22, 3690, 2049, 1247, 5432} accept # handle 13
                 reject with icmp type net-unreachable # handle 14
        }

        chain forward {
                 type filter hook forward priority -300;
                 udp sport { 67, 68} reject # handle 16
                 udp dport { 67, 68} reject # handle 17
                 ip daddr 10.2.3.255 reject # handle 18
                 oif { tap0, tap1} reject # handle 22
                 iif { tap0, tap1} reject # handle 23
        }
}


localhost ~ # nft list table routing -nnna
table ip routing {
        chain forward {
                 type filter hook forward priority -300;
                 udp sport { 68, 67} reject # handle 2
                 udp dport { 68, 67} reject # handle 3
                 ip daddr 10.2.3.255 reject # handle 4
                 iif { tap1, tap0} reject # handle 5
                 oif { tap0, tap1} reject # handle 6
        }
}


Last edited by rickvernam on Wed Jul 08, 2015 3:17 pm; edited 1 time in total
Back to top
View user's profile Send private message
rickvernam
Guru
Guru


Joined: 09 Jul 2004
Posts: 310

PostPosted: Wed Jul 08, 2015 3:17 pm    Post subject: Reply with quote

Turns out that QEmu, when network is specified via "-net ...", will create an internal software HUB where incoming traffic is repeated across all other "-net ..." devices.

A depracated solution is to separate the devices via vlan configuration option "-net ...,vlan=0" and "-net ...,vlan=1" - this is not Virtual Lan in the traditional sense, but rather an internal QEmu configuration to separate the network devices.

A suggested solution is to switch away from using "-net ..." and instead use "-netdev ... -device ...."

I discovered this, along with a few more details than what I've reposted here, on this email thread: http://lists.gnu.org/archive/html/qemu-discuss/2014-06/msg00067.html
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum