Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
SOLVED: NF Tables Router + VM
View unanswered posts
View posts from last 24 hours

Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message

Joined: 09 Jul 2004
Posts: 310

PostPosted: Thu Jun 04, 2015 7:45 pm    Post subject: SOLVED: NF Tables Router + VM Reply with quote

I have a setup that I've used successfully with kernel 3.13 through 3.17 on a router within my LAN.

                           |                                                              |
                           |   br1 (                    ( br0        |             |   / --tap0--( \         |
---------------------------|---                                                  ---------|----------
                           |   \ --enp11s0                                      /         |
                           |                                                  enp4s0f0    |
                           |                                                  enp3s0f0    |
default gateway            |                                                  enp3s0f1    |
---------------------------|--- enp4s0f1 (wan ip address)                                 |

The router is connected to three networks
1 - directly to a modem to an ISP. This is the "Router's" default gateway.
2 - to a network. "Router" is the gateway for the network. It provides dhcp & some other network services.
3 - to a network, which has some other stuff and also it's own gateway through a different ISP. "Router" routes some destinations through this network instead of the default gateway. "Router" also does NAT for stuff going into the network.

"Router" runs a QEmu virtual machine for a network appliance which needs to be accessible from both the network & the network. Therefore, I have two tun/tap interfaces connected to the VM; each tap interface is then bridged to the appropriate interface.
QEmu provides two network interfaces to the VM via:
-net tap,ifname=tap0,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:01
-net tap,ifname=tap1,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:02

The problem I'm experiencing...
Starting with kernel 3.18, but only when the VM is running, dhcp traffic from the network ends up on the network, despite nftables rules in "Router" to block that traffic from leaving the network - these same rules worked in previous linux versions from 3.13 to 3.17.
I have tcpdump packet captures on the dhcp server with dhcp packets containing source, whereas other traffic (legitimately) originating in the network is correctly NAT'ed to show source address.
I don't believe the guest OS within the VM has anything to do with this; I continue to experience the problem when I run the VM without an HD (ie, no OS running inside the VM).
As you might expect if I remove either tap device from it's bridge, the problem ceases. Once I add the tap device back into it's bridge, it starts back up again.
I am trying to prevent that dhcp traffic from "leaking" onto the netowrk; ie, the has it's own dhcp server, and dhcp traffic should not traverse between the two networks. Any thoughts as to why this might have started and/or how I may fix the problem?

"Router's" /etc/conf.d/net file is like so (wan ip addresses edited to a.b.c.d, e.f.g.h & i.j.k.l):
# ===========================================================
# tap0 is used by VM Appliance on the network
# ===========================================================


bridge_br1="enp11s0 tap0"
rc_net_br1_need="net.enp11s0 net.tap0"
routes_br1=" dev br1
a.b.c.d/29 via
e.f.g.h/29 via via via"

# ===========================================================
# tap1 is used by VM Appliance on the network
# ===========================================================



bridge_br0="enp3s0f0 enp3s0f1 enp4s0f0 tap1"
rc_net_br0_need="net.enp3s0f0 net.enp3s0f1 net.enp4s0f0 net.tap1"

# ===========================================================
# This port will go down the secondary ISP pipe and become the
# "primary" gateway.  It will not bridge with anything.
# ===========================================================

config_enp4s0f1="i.j.k.l netmask"
routes_enp4s0f1="default via m.n.o.p"

Bridge info:
localhost ~ # brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.5254001201ff       no              enp3s0f0
br1             8000.5254001201fe       no              enp11s0

NF Tables rules:
Initially, starting with 3.13 through 3.17, I did not have the routing nf table & the filter forward chain was just dport 67 & priority 0. The additional rules in the filter forward chain & the routing table were attempts to stop this "leakage."
localhost ~ # nft list tables
table nat
table filter
table routing

localhost ~ # nft list table nat -nnna
table ip nat {
        chain post {
                 type nat hook postrouting priority 0;
                 ip saddr ip daddr a.b.c.d/29 oif br1 snat # handle 2
                 ip saddr ip daddr oif br1 snat # handle 3
                 ip saddr ip daddr oif br1 snat # handle 4
                 ip saddr ip daddr oif br1 snat # handle 5
                 ip saddr ip daddr e.f.g.h/29 oif br1 snat # handle 6
                 ip saddr oif enp4s0f1 snat i.j.k.l # handle 7

        chain pre {
                 type nat hook prerouting priority 0;
                  (edited out for brevity)

localhost ~ # nft list table filter -nnna
table ip filter {
        chain incoming {
                 type filter hook input priority 0;
                 tcp flags & (fin | syn) == fin | syn drop # handle 2
                 tcp flags & (syn | rst) == syn | rst drop # handle 3
                 tcp flags & (fin | syn | rst | psh | ack | urg) < fin drop # handle 4
                 tcp flags & (fin | syn | rst | psh | ack | urg) == fin | psh | urg drop # handle 5
                 ct state { related, established} accept # handle 6
                 ct state invalid drop # handle 7
                 iifname "lo" accept # handle 8
                 ip saddr accept # handle 9
                 ip saddr a.b.c.d/29 tcp dport { 2049, 3690, 22, 5432} accept # handle 10
                 ip saddr tcp dport { 2049, 22, 1247, 3690, 5432} accept # handle 11
                 ip saddr tcp dport { 3690, 5432, 1247, 2049, 22} accept # handle 12
                 ip saddr tcp dport { 22, 3690, 2049, 1247, 5432} accept # handle 13
                 reject with icmp type net-unreachable # handle 14

        chain forward {
                 type filter hook forward priority -300;
                 udp sport { 67, 68} reject # handle 16
                 udp dport { 67, 68} reject # handle 17
                 ip daddr reject # handle 18
                 oif { tap0, tap1} reject # handle 22
                 iif { tap0, tap1} reject # handle 23

localhost ~ # nft list table routing -nnna
table ip routing {
        chain forward {
                 type filter hook forward priority -300;
                 udp sport { 68, 67} reject # handle 2
                 udp dport { 68, 67} reject # handle 3
                 ip daddr reject # handle 4
                 iif { tap1, tap0} reject # handle 5
                 oif { tap0, tap1} reject # handle 6

Last edited by rickvernam on Wed Jul 08, 2015 3:17 pm; edited 1 time in total
Back to top
View user's profile Send private message

Joined: 09 Jul 2004
Posts: 310

PostPosted: Wed Jul 08, 2015 3:17 pm    Post subject: Reply with quote

Turns out that QEmu, when network is specified via "-net ...", will create an internal software HUB where incoming traffic is repeated across all other "-net ..." devices.

A depracated solution is to separate the devices via vlan configuration option "-net ...,vlan=0" and "-net ...,vlan=1" - this is not Virtual Lan in the traditional sense, but rather an internal QEmu configuration to separate the network devices.

A suggested solution is to switch away from using "-net ..." and instead use "-netdev ... -device ...."

I discovered this, along with a few more details than what I've reposted here, on this email thread:
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum