View previous topic :: View next topic |
Author |
Message |
viacheslavg n00b
Joined: 23 Sep 2016 Posts: 8
|
Posted: Tue Aug 31, 2021 2:25 pm Post subject: [SOLVED] linux bridge not forwarding traffic outside |
|
|
I'm trying to setup network namespace which later will be used by dedicated user. The idea is that this user will play with network configurations (e.g. setup VPN) but this changes should not affect network on host (i.e. "default" network namespace).
What I did so far:
1. Setup network namespace along with veth pair:
Code: |
ip netns add vnet0
ip link add veth0 type veth peer name eth0 netns vnet0
|
2. Configure interfaces and routes in namespace:
Code: |
ip netns exec vnet0 ip link set lo up
ip netns exec vnet0 ip addr add 192.168.8.2/24 dev eth0
ip netns exec vnet0 ip link set eth0 up
ip netns exec vnet0 ip route add default via 192.168.8.1 dev eth0
|
3. Create bridge in the host and add veth there:
Code: |
ip link add name br0 type bridge
ip addr add 192.168.8.1/24 brd 192.168.8.255 dev br0
ip link set veth0 master br0
ip link set br0 up
|
As result I have following on host:
Code: |
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2e:9e:3f:99:fc:c9 brd ff:ff:ff:ff:ff:ff
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
4: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether e4:7f:b2:17:26:49 brd ff:ff:ff:ff:ff:ff
5: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 80:19:34:c0:e6:8a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.120/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
valid_lft 41981sec preferred_lft 41981sec
inet6 fd63:61e4:95ff::f97/128 scope global dynamic noprefixroute
valid_lft 85184sec preferred_lft 85184sec
inet6 fd63:61e4:95ff:0:1cdd:1f29:e279:ca55/64 scope global dynamic noprefixroute
valid_lft 5984sec preferred_lft 584sec
inet6 fe80::5957:17e5:842f:3cd8/64 scope link noprefixroute
valid_lft forever preferred_lft forever
8: veth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff link-netns vnet0
inet6 fe80::d04:c570:6e2d:fdc3/64 scope link noprefixroute
valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.8.1/24 brd 192.168.8.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::acde:a7ff:fe6d:ddd/64 scope link
valid_lft forever preferred_lft forever
# ip route
default via 192.168.1.1 dev wlp3s0 proto dhcp metric 600
192.168.1.0/24 dev wlp3s0 proto kernel scope link src 192.168.1.120 metric 600
192.168.8.0/24 dev br0 proto kernel scope link src 192.168.8.1
|
and following inside network namespace:
Code: |
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3a:d2:8c:56:cc:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.8.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::38d2:8cff:fe56:cc6b/64 scope link
valid_lft forever preferred_lft forever
# ip route
default via 192.168.8.1 dev eth0
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.2
|
My problem is that I'm not able to reach any IP from within this namespace outside of host.
e.g.:
Code: |
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2082ms
|
What I'm doing wrong? Could it be that packets are somehow filtered in bridge? I have no special rules in iptables:
Code: |
# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
# ebtables -t filter -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# ebtables -t nat -L
Bridge table: nat
Bridge chain: PREROUTING, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT
|
bridge also looks good:
Code: |
# brctl showstp br0
br0
bridge id 8000.aedea76d0ddd
designated root 8000.aedea76d0ddd
root port 0 path cost 0
max age 20.00 bridge max age 20.00
hello time 2.00 bridge hello time 2.00
forward delay 15.00 bridge forward delay 15.00
ageing time 300.00
hello timer 0.00 tcn timer 0.00
topology change timer 0.00 gc timer 201.75
flags
veth0 (1)
port id 8001 state forwarding
designated root 8000.aedea76d0ddd path cost 2
designated bridge 8000.aedea76d0ddd message age timer 0.00
designated port 8001 forward delay timer 0.00
designated cost 0 hold timer 0.00
flags
|
As an additional observation: if i start docker, which will create docker0 bridge and bunch of rules in iptables and add my veth0 interface to docker0 bridge then everything works fine. I have no glue why docker0 bridge works fine but mine not.
Any idea?
Last edited by viacheslavg on Fri Sep 03, 2021 11:19 am; edited 1 time in total |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3095
|
Posted: Tue Aug 31, 2021 3:01 pm Post subject: |
|
|
Depending on your network, you should either:
* enable ip forwarding (via sysctl) and add MASQUERADE rule to POSTROUTING in nat table - most common.
* bridge your veth with a physical devices (add your wifi to the bridge and move wifi's IP there too).
* (rare - if you don't know this is your use case, it is not) just enable ip forwarding. |
|
Back to top |
|
|
viacheslavg n00b
Joined: 23 Sep 2016 Posts: 8
|
Posted: Wed Sep 01, 2021 3:46 pm Post subject: |
|
|
Thanks for helping.
As I mentioned earlier everything works out of the box if i move my veth to the docker0 bridge (created by docker). I don't need to put wifi to the bridge enable IP forwarding etc.
I replicated all the iptable rules (which docker creates for docker0) for my bridge (including MASQUERADE you mention). so now I have:
Code: | # iptables -t nat -v -L
Chain PREROUTING (policy ACCEPT 23 packets, 3192 bytes)
pkts bytes target prot opt in out source destination
26 3130 DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 5 packets, 1432 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 424 packets, 28043 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- any any anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 424 packets, 28043 bytes)
pkts bytes target prot opt in out source destination
3 252 MASQUERADE all -- any !docker0 192.168.64.0/24 anywhere
0 0 MASQUERADE all -- any !br-68c3e5c395af 192.168.70.0/24 anywhere
0 0 MASQUERADE all -- any !br0 192.168.8.0/24 anywhere
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 any anywhere anywhere
0 0 RETURN all -- br-68c3e5c395af any anywhere anywhere
0 0 RETURN all -- br0 any anywhere anywhere
# iptables -t filter -v -L
Chain INPUT (policy ACCEPT 1152 packets, 459K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 4 packets, 336 bytes)
pkts bytes target prot opt in out source destination
80 6664 DOCKER-USER all -- any any anywhere anywhere
80 6664 DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
13 1092 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
9 756 DOCKER all -- any docker0 anywhere anywhere
13 1092 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
0 0 ACCEPT all -- any br-68c3e5c395af anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-68c3e5c395af anywhere anywhere
0 0 ACCEPT all -- br-68c3e5c395af !br-68c3e5c395af anywhere anywhere
0 0 ACCEPT all -- br-68c3e5c395af br-68c3e5c395af anywhere anywhere
0 0 ACCEPT all -- any br0 anywhere anywhere
0 0 DOCKER all -- any br0 anywhere anywhere
0 0 ACCEPT all -- br0 !br0 anywhere anywhere
0 0 ACCEPT all -- br0 br0 anywhere anywhere
Chain OUTPUT (policy ACCEPT 1230 packets, 160K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (3 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- br0 !br0 anywhere anywhere
13 1092 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 anywhere anywhere
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-68c3e5c395af !br-68c3e5c395af anywhere anywhere
80 6664 RETURN all -- any any anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (3 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any br0 anywhere anywhere
0 0 DROP all -- any docker0 anywhere anywhere
0 0 DROP all -- any br-68c3e5c395af anywhere anywhere
13 1092 RETURN all -- any any anywhere anywhere
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
80 6664 RETURN all -- any any anywhere anywhere
|
still no go. I suspect docker does yet more "magic" with it's docker0 bridge. But what else could it be? no idea....maybe some extra netfilter table I'm not aware of? |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3095
|
Posted: Wed Sep 01, 2021 4:23 pm Post subject: |
|
|
If you showed us the result of iptables-save instead of iptables -vL, it would be easier to follow.
It provides an unambiguous dump (one that can be imported back). It will also include all tables without any need to manually specify them. And I suppose many people here are more used to its format (yes, including me).
Docker is a really messy piece of software. It alters your firewall rules in weird ways, it can disable your firewall, drop some traffic you actually wanted, bang your milk or drink your cat.
If you want to run it alongside anything else on the same host, you should probably isolate it with another network namespace. It can be done and is not very hard once you know how to use namespaces (AFAIK it breaks swarm mode, but pretty much any other docker manager will work, should you need it) so you're almost there: just create another ns, connect it to the host with a pair of veth and run docker daemon inside. And either bridge or route traffic as needed.
This basically requires you to write your own init scripts; but it's still not very hard to do. |
|
Back to top |
|
|
viacheslavg n00b
Joined: 23 Sep 2016 Posts: 8
|
Posted: Fri Sep 03, 2021 11:18 am Post subject: |
|
|
Indeed iptables-save output is much better and it helped me to sort out the problem. Docker was also involved but I can't say it does silly things.
Here is explanation:
I had /etc/sysctl.d/local.conf with
Code: | net.ipv4.ip_forward=1 |
so, I thought ip forwarding is enabled, but as it appears in /etc/sysctl.conf I had:
Code: | net.ipv4.ip_forward=0 |
and as result forwarding was disabled. Which is strange, my assumption was that custom configs in sysctl.d/ should override the one from /etc/sysctl.conf
However when docker starts it enables this option and to preserve existing behavior for the rest of system it changes FORWARD policy from ACCEPT to DROP (plus of course adds rules for it's own bridge to make it working).
Thus when I start my system w/o docker enabled forwarding was not working because of net.ipv4.ip_forward=0 setting, but when I start my system with docker forwarding was disabled for my bridge in iptables by docker.
Summarizing, the solution is following:
Possible two options:
1. If you are OK with having net.ipv4.ip_forward=1 system wide then only following rule required in iptables:
Code: | iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -j MASQUERADE |
2. If you need to have net.ipv4.ip_forward=0 and docker is installed then two additional rules can fix the problem:
Code: |
iptables -A FORWARD -o br0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 ! -o br0 -j ACCEPT
|
Thanks a lot for helping! |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|