| View previous topic :: View next topic |
| Author |
Message |
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
 Posted: Thu Jun 04, 2015 7:45 pm Post subject: SOLVED: NF Tables Router + VM |
 |
|
I have a setup that I've used successfully with kernel 3.13 through 3.17 on a router within my LAN.
| Code: | "Router"
________________________________________________________________
| |
| br1 (192.168.1.7) (10.2.2.1) br0 |
192.168.1.0/24 | / --tap0--(192.168.1.24)--VM--(10.2.2.2)--tap1-- \ |
---------------------------|--- ---------|---------- 10.2.2.0/23
| \ --enp11s0 / |
| enp4s0f0 |
| enp3s0f0 |
default gateway | enp3s0f1 |
---------------------------|--- enp4s0f1 (wan ip address) |
|--------------------------------------------------------------| |
The router is connected to three networks
1 - directly to a modem to an ISP. This is the "Router's" default gateway.
2 - to a 10.2.2.0/23 network. "Router" is the gateway for the 10.2.2.0/23 network. It provides dhcp & some other network services.
3 - to a 192.168.1.0/24 network, which has some other stuff and also it's own gateway through a different ISP. "Router" routes some destinations through this network instead of the default gateway. "Router" also does NAT for stuff going into the 192.168.1.0/24 network.
"Router" runs a QEmu virtual machine for a network appliance which needs to be accessible from both the 192.168.1.0/24 network & the 10.2.2.0/23 network. Therefore, I have two tun/tap interfaces connected to the VM; each tap interface is then bridged to the appropriate interface.
QEmu provides two network interfaces to the VM via:
| Code: | -net tap,ifname=tap0,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:01
-net tap,ifname=tap1,script=no,downscript=no -net nic,model=virtio,macaddr=02:00:00:00:00:02 |
The problem I'm experiencing...
Starting with kernel 3.18, but only when the VM is running, dhcp traffic from the 10.2.2.0/23 network ends up on the 192.168.1.0/24 network, despite nftables rules in "Router" to block that traffic from leaving the 10.2.2.0/23 network - these same rules worked in previous linux versions from 3.13 to 3.17.
I have tcpdump packet captures on the 192.168.1.0/24 dhcp server with dhcp packets containing 10.2.2.0/23 source, whereas other traffic (legitimately) originating in the 10.2.2.0/23 network is correctly NAT'ed to show 192.168.1.7 source address.
I don't believe the guest OS within the VM has anything to do with this; I continue to experience the problem when I run the VM without an HD (ie, no OS running inside the VM).
As you might expect if I remove either tap device from it's bridge, the problem ceases. Once I add the tap device back into it's bridge, it starts back up again.
I am trying to prevent that dhcp traffic from "leaking" onto the 192.168.1.0/24 netowrk; ie, the 10.2.2.0/23 has it's own dhcp server, and dhcp traffic should not traverse between the two networks. Any thoughts as to why this might have started and/or how I may fix the problem?
"Router's" /etc/conf.d/net file is like so (wan ip addresses edited to a.b.c.d, e.f.g.h & i.j.k.l):
| Code: | # ===========================================================
# tap0 is used by VM Appliance on the 192.168.1.0/24 network
# ===========================================================
tuntap_tap0="tap"
config_tap0="null"
config_enp11s0="null"
config_br1="192.168.1.7/24"
bridge_br1="enp11s0 tap0"
rc_net_br1_need="net.enp11s0 net.tap0"
mac_br1="52:54:00:12:01:fe"
routes_br1=" 192.168.1.0/24 dev br1
a.b.c.d/29 via 192.168.1.1
e.f.g.h/29 via 192.168.1.1
192.168.2.0/24 via 192.168.1.1
192.168.4.0/24 via 192.168.1.1"
# ===========================================================
# tap1 is used by VM Appliance on the 10.2.2.0/23 network
# ===========================================================
config_enp3s0f0="null"
config_enp3s0f1="null"
config_enp4s0f0="null"
tuntap_tap1="tap"
config_tap1="null"
config_br0="10.2.2.1/23"
bridge_br0="enp3s0f0 enp3s0f1 enp4s0f0 tap1"
rc_net_br0_need="net.enp3s0f0 net.enp3s0f1 net.enp4s0f0 net.tap1"
mac_br0="52:54:00:12:01:ff"
# ===========================================================
# This port will go down the secondary ISP pipe and become the
# "primary" gateway. It will not bridge with anything.
# ===========================================================
config_enp4s0f1="i.j.k.l netmask 255.255.255.248"
routes_enp4s0f1="default via m.n.o.p" |
Bridge info:
| Code: | localhost ~ # brctl show
bridge name bridge id STP enabled interfaces
br0 8000.5254001201ff no enp3s0f0
enp3s0f1
enp4s0f0
tap1
br1 8000.5254001201fe no enp11s0
tap0 |
NF Tables rules:
Initially, starting with 3.13 through 3.17, I did not have the routing nf table & the filter forward chain was just dport 67 & priority 0. The additional rules in the filter forward chain & the routing table were attempts to stop this "leakage."
| Code: | localhost ~ # nft list tables
table nat
table filter
table routing
localhost ~ # nft list table nat -nnna
table ip nat {
chain post {
type nat hook postrouting priority 0;
ip saddr 10.2.2.0/23 ip daddr a.b.c.d/29 oif br1 snat 192.168.1.7 # handle 2
ip saddr 10.2.2.0/23 ip daddr 192.168.1.0/24 oif br1 snat 192.168.1.7 # handle 3
ip saddr 10.2.2.0/23 ip daddr 192.168.2.0/24 oif br1 snat 192.168.1.7 # handle 4
ip saddr 10.2.2.0/23 ip daddr 192.168.4.0/24 oif br1 snat 192.168.1.7 # handle 5
ip saddr 10.2.2.0/23 ip daddr e.f.g.h/29 oif br1 snat 192.168.1.7 # handle 6
ip saddr 10.2.2.0/23 oif enp4s0f1 snat i.j.k.l # handle 7
}
chain pre {
type nat hook prerouting priority 0;
(edited out for brevity)
}
}
localhost ~ # nft list table filter -nnna
table ip filter {
chain incoming {
type filter hook input priority 0;
tcp flags & (fin | syn) == fin | syn drop # handle 2
tcp flags & (syn | rst) == syn | rst drop # handle 3
tcp flags & (fin | syn | rst | psh | ack | urg) < fin drop # handle 4
tcp flags & (fin | syn | rst | psh | ack | urg) == fin | psh | urg drop # handle 5
ct state { related, established} accept # handle 6
ct state invalid drop # handle 7
iifname "lo" accept # handle 8
ip saddr 10.2.2.0/23 accept # handle 9
ip saddr a.b.c.d/29 tcp dport { 2049, 3690, 22, 5432} accept # handle 10
ip saddr 192.168.4.0/24 tcp dport { 2049, 22, 1247, 3690, 5432} accept # handle 11
ip saddr 192.168.2.0/24 tcp dport { 3690, 5432, 1247, 2049, 22} accept # handle 12
ip saddr 192.168.1.0/24 tcp dport { 22, 3690, 2049, 1247, 5432} accept # handle 13
reject with icmp type net-unreachable # handle 14
}
chain forward {
type filter hook forward priority -300;
udp sport { 67, 68} reject # handle 16
udp dport { 67, 68} reject # handle 17
ip daddr 10.2.3.255 reject # handle 18
oif { tap0, tap1} reject # handle 22
iif { tap0, tap1} reject # handle 23
}
}
localhost ~ # nft list table routing -nnna
table ip routing {
chain forward {
type filter hook forward priority -300;
udp sport { 68, 67} reject # handle 2
udp dport { 68, 67} reject # handle 3
ip daddr 10.2.3.255 reject # handle 4
iif { tap1, tap0} reject # handle 5
oif { tap0, tap1} reject # handle 6
}
} |
Last edited by rickvernam on Wed Jul 08, 2015 3:17 pm; edited 1 time in total |
|
| Back to top |
|
 |
rickvernam
Guru
Joined: 09 Jul 2004
Posts: 313
|
| Posted: Wed Jul 08, 2015 3:17 pm Post subject: |
|
|
Turns out that QEmu, when network is specified via "-net ...", will create an internal software HUB where incoming traffic is repeated across all other "-net ..." devices.
A depracated solution is to separate the devices via vlan configuration option "-net ...,vlan=0" and "-net ...,vlan=1" - this is not Virtual Lan in the traditional sense, but rather an internal QEmu configuration to separate the network devices.
A suggested solution is to switch away from using "-net ..." and instead use "-netdev ... -device ...."
I discovered this, along with a few more details than what I've reposted here, on this email thread: http://lists.gnu.org/archive/html/qemu-discuss/2014-06/msg00067.html |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
