[SOLVED] linux bridge not forwarding traffic outside

[viacheslavg]

I'm trying to setup network namespace which later will be used by dedicated user. The idea is that this user will play with network configurations (e.g. setup VPN) but this changes should not affect network on host (i.e. "default" network namespace).
What I did so far:
1. Setup network namespace along with veth pair:

Code: Select all

ip netns add vnet0
ip link add veth0 type veth peer name eth0 netns vnet0

2. Configure interfaces and routes in namespace:

Code: Select all

ip netns exec vnet0 ip link set lo up
ip netns exec vnet0 ip addr add 192.168.8.2/24 dev eth0
ip netns exec vnet0 ip link set eth0 up
ip netns exec vnet0 ip route add default via 192.168.8.1 dev eth0

3. Create bridge in the host and add veth there:

Code: Select all

ip link add name br0 type bridge
ip addr add 192.168.8.1/24 brd 192.168.8.255 dev br0
ip link set veth0 master br0
ip link set br0 up

As result I have following on host:

Code: Select all

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2e:9e:3f:99:fc:c9 brd ff:ff:ff:ff:ff:ff
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether e4:7f:b2:17:26:49 brd ff:ff:ff:ff:ff:ff
5: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 80:19:34:c0:e6:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 41981sec preferred_lft 41981sec
    inet6 fd63:61e4:95ff::f97/128 scope global dynamic noprefixroute 
       valid_lft 85184sec preferred_lft 85184sec
    inet6 fd63:61e4:95ff:0:1cdd:1f29:e279:ca55/64 scope global dynamic noprefixroute 
       valid_lft 5984sec preferred_lft 584sec
    inet6 fe80::5957:17e5:842f:3cd8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
8: veth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff link-netns vnet0
    inet6 fe80::d04:c570:6e2d:fdc3/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.1/24 brd 192.168.8.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::acde:a7ff:fe6d:ddd/64 scope link 
       valid_lft forever preferred_lft forever

# ip route
default via 192.168.1.1 dev wlp3s0 proto dhcp metric 600 
192.168.1.0/24 dev wlp3s0 proto kernel scope link src 192.168.1.120 metric 600 
192.168.8.0/24 dev br0 proto kernel scope link src 192.168.8.1 

and following inside network namespace:

Code: Select all

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:d2:8c:56:cc:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.8.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::38d2:8cff:fe56:cc6b/64 scope link 
       valid_lft forever preferred_lft forever

# ip route
default via 192.168.8.1 dev eth0 
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.2 

My problem is that I'm not able to reach any IP from within this namespace outside of host.
e.g.:

Code: Select all

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2082ms

What I'm doing wrong? Could it be that packets are somehow filtered in bridge? I have no special rules in iptables:

Code: Select all

# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

# ebtables -t filter -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

# ebtables -t nat -L
Bridge table: nat

Bridge chain: PREROUTING, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT

bridge also looks good:

Code: Select all

# brctl showstp br0
br0
 bridge id		8000.aedea76d0ddd
 designated root	8000.aedea76d0ddd
 root port		   0			path cost		   0
 max age		  20.00			bridge max age		  20.00
 hello time		   2.00			bridge hello time	   2.00
 forward delay		  15.00			bridge forward delay	  15.00
 ageing time		 300.00
 hello timer		   0.00			tcn timer		   0.00
 topology change timer	   0.00			gc timer		 201.75
 flags			


veth0 (1)
 port id		8001			state		     forwarding
 designated root	8000.aedea76d0ddd	path cost		   2
 designated bridge	8000.aedea76d0ddd	message age timer	   0.00
 designated port	8001			forward delay timer	   0.00
 designated cost	   0			hold timer		   0.00
 flags			

As an additional observation: if i start docker, which will create docker0 bridge and bunch of rules in iptables and add my veth0 interface to docker0 bridge then everything works fine. I have no glue why docker0 bridge works fine but mine not.
Any idea?

[szatox]

Depending on your network, you should either:
* enable ip forwarding (via sysctl) and add MASQUERADE rule to POSTROUTING in nat table - most common.
* bridge your veth with a physical devices (add your wifi to the bridge and move wifi's IP there too).
* (rare - if you don't know this is your use case, it is not) just enable ip forwarding.

[viacheslavg]

Thanks for helping.
As I mentioned earlier everything works out of the box if i move my veth to the docker0 bridge (created by docker). I don't need to put wifi to the bridge enable IP forwarding etc.
I replicated all the iptable rules (which docker creates for docker0) for my bridge (including MASQUERADE you mention). so now I have:

Code: Select all

# iptables -t nat -v -L
Chain PREROUTING (policy ACCEPT 23 packets, 3192 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   26  3130 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 5 packets, 1432 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 424 packets, 28043 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 424 packets, 28043 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    3   252 MASQUERADE  all  --  any    !docker0  192.168.64.0/24      anywhere            
    0     0 MASQUERADE  all  --  any    !br-68c3e5c395af  192.168.70.0/24      anywhere            
    0     0 MASQUERADE  all  --  any    !br0    192.168.8.0/24       anywhere            

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
    0     0 RETURN     all  --  br-68c3e5c395af any     anywhere             anywhere            
    0     0 RETURN     all  --  br0    any     anywhere             anywhere            

# iptables -t filter -v -L
Chain INPUT (policy ACCEPT 1152 packets, 459K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 4 packets, 336 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   80  6664 DOCKER-USER  all  --  any    any     anywhere             anywhere            
   80  6664 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere            
   13  1092 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    9   756 DOCKER     all  --  any    docker0  anywhere             anywhere            
   13  1092 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    br-68c3e5c395af  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    br-68c3e5c395af  anywhere             anywhere            
    0     0 ACCEPT     all  --  br-68c3e5c395af !br-68c3e5c395af  anywhere             anywhere            
    0     0 ACCEPT     all  --  br-68c3e5c395af br-68c3e5c395af  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    br0     anywhere             anywhere            
    0     0 DOCKER     all  --  any    br0     anywhere             anywhere            
    0     0 ACCEPT     all  --  br0    !br0    anywhere             anywhere            
    0     0 ACCEPT     all  --  br0    br0     anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 1230 packets, 160K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (3 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br0    !br0    anywhere             anywhere            
   13  1092 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere            
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-68c3e5c395af !br-68c3e5c395af  anywhere             anywhere            
   80  6664 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    br0     anywhere             anywhere            
    0     0 DROP       all  --  any    docker0  anywhere             anywhere            
    0     0 DROP       all  --  any    br-68c3e5c395af  anywhere             anywhere            
   13  1092 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   80  6664 RETURN     all  --  any    any     anywhere             anywhere            

still no go. I suspect docker does yet more "magic" with it's docker0 bridge. But what else could it be? no idea....maybe some extra netfilter table I'm not aware of?

[szatox]

If you showed us the result of iptables-save instead of iptables -vL, it would be easier to follow.
It provides an unambiguous dump (one that can be imported back). It will also include all tables without any need to manually specify them. And I suppose many people here are more used to its format (yes, including me).

Docker is a really messy piece of software. It alters your firewall rules in weird ways, it can disable your firewall, drop some traffic you actually wanted, bang your milk or drink your cat.
If you want to run it alongside anything else on the same host, you should probably isolate it with another network namespace. It can be done and is not very hard once you know how to use namespaces (AFAIK it breaks swarm mode, but pretty much any other docker manager will work, should you need it) so you're almost there: just create another ns, connect it to the host with a pair of veth and run docker daemon inside. And either bridge or route traffic as needed.
This basically requires you to write your own init scripts; but it's still not very hard to do.

[viacheslavg]

Indeed iptables-save output is much better and it helped me to sort out the problem. Docker was also involved but I can't say it does silly things.
Here is explanation:
I had /etc/sysctl.d/local.conf with

Code: Select all

net.ipv4.ip_forward=1

so, I thought ip forwarding is enabled, but as it appears in /etc/sysctl.conf I had:

Code: Select all

net.ipv4.ip_forward=0

and as result forwarding was disabled. Which is strange, my assumption was that custom configs in sysctl.d/ should override the one from /etc/sysctl.conf
However when docker starts it enables this option and to preserve existing behavior for the rest of system it changes FORWARD policy from ACCEPT to DROP (plus of course adds rules for it's own bridge to make it working).
Thus when I start my system w/o docker enabled forwarding was not working because of net.ipv4.ip_forward=0 setting, but when I start my system with docker forwarding was disabled for my bridge in iptables by docker.
Summarizing, the solution is following:
Possible two options:
1. If you are OK with having net.ipv4.ip_forward=1 system wide then only following rule required in iptables:

Code: Select all

iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -j MASQUERADE

2. If you need to have net.ipv4.ip_forward=0 and docker is installed then two additional rules can fix the problem:

Code: Select all

iptables -A FORWARD -o br0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 ! -o br0 -j ACCEPT

Thanks a lot for helping!