setting up a gentoo router and two subnets [SOLVED]

Message

frank0x01 · Post by **frank0x01** » Thu Apr 06, 2023 10:34 am

Hello,

I'm trying to set up a router that between 2 subnets. The router is a bananapi-r2 running gentoo with the following configuration:

Code: Select all

ip a:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ce:04:b7:4b:85:b1 brd ff:ff:ff:ff:ff:ff
    inet 169.254.200.153/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.255.255/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::cc04:b7ff:fe4b:85b1/64 scope link
       valid_lft forever preferred_lft forever
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:b0:fc:04:a1:73 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global wan
       valid_lft forever preferred_lft forever
    inet6 2a02:a448:7890:1:40b0:fcff:fe04:a173/64 scope global dynamic mngtmpaddr
       valid_lft 258723sec preferred_lft 172323sec
    inet6 fe80::40b0:fcff:fe04:a173/64 scope link
       valid_lft forever preferred_lft forever
5: lan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:c3:0b:dd:07:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global lan0
       valid_lft forever preferred_lft forever
    inet6 2a02:a448:7890:100:cc3:bff:fedd:784/64 scope global dynamic mngtmpaddr
       valid_lft 201625sec preferred_lft 115225sec
    inet6 fe80::cc3:bff:fedd:784/64 scope link
       valid_lft forever preferred_lft forever
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ce:04:b7:4b:85:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global lan1
       valid_lft forever preferred_lft forever
    inet6 fe80::cc04:b7ff:fe4b:85b1/64 scope link
       valid_lft forever preferred_lft forever
7: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ce:04:b7:4b:85:b1 brd ff:ff:ff:ff:ff:ff
8: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ce:04:b7:4b:85:b1 brd ff:ff:ff:ff:ff:ff
9: br0: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 66:d4:30:f0:ea:d3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.1/32 brd 10.0.10.255 scope global br0
       valid_lft forever preferred_lft forever

ip r:

default via 192.168.2.254 dev wan metric 4
default via 192.168.1.1 dev lan0 metric 5
default via 10.0.1.1 dev lan1 metric 6
default via 10.0.10.1 dev br0 metric 9 linkdown
10.0.1.0/24 dev lan1 proto kernel scope link src 10.0.1.1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.200.153
192.168.1.0/24 dev lan0 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev wan proto kernel scope link src 192.168.2.1

It has access to the internet via the wan interface. Furthermore it is connected via lan0 to a linksys router (for the moment) with the rest of my local network. I can ping all active devices on the bpi-r2 router from my local network.

On lan1 I have a computer connected with the following configuration:

Code: Select all

ip a:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b2:c9:97:0b:f8:00 brd ff:ff:ff:ff:ff:ff
    inet 169.254.172.112/16 brd 169.254.255.255 scope global noprefixroute dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::e5dd:4e36:2a24:d268/64 scope link
       valid_lft forever preferred_lft forever
3: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:13:3b:0f:37:be brd ff:ff:ff:ff:ff:ff
4: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:13:3b:0f:37:bf brd ff:ff:ff:ff:ff:ff
5: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 90:2b:34:6d:0e:f6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.23/16 brd 10.0.1.255 scope global dynamic noprefixroute enp7s0
       valid_lft 248586sec preferred_lft 217294sec
    inet6 fe80::4605:614b:e24c:f634/64 scope link
       valid_lft forever preferred_lft forever
6: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:c3:55:43:88 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

ip r:

default via 10.0.1.1 dev enp7s0 proto dhcp src 10.0.1.23 metric 1005
10.0.0.0/16 dev enp7s0 proto dhcp scope link src 10.0.1.23 metric 1005
169.254.0.0/16 dev dummy0 scope link src 169.254.172.112 metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

Again, i can ping all active devices on the bpi-r2 router from this last machine, the subnets don't see each other, however. What I want is that I can access both subnets from each other and give them access to the wan gateway on the bpi-r2 router. Can anyone point me in the right direction?

Edit: additional info.
I have a static route on the linksys router: 10.0.0.0/16 gateway 192.168.1.2 and have enabled ipv4/ip_forward on the bpi-r2. Both the bpi-r2 and the other machine have no iptables configuration.

alamahant · Post by **alamahant** » Thu Apr 06, 2023 6:03 pm

Welcome to Gentoo Forums!
I see in router machine you have 4 default routes.Keep only the one connecting to the internet and use routing for the lans
On lan0 machine add route

Code: Select all

ip route add 10.0.0.0/16 via 192.168.1.2
ip route add 192.168.2.0/24 via 192.168.1.2

On lan1 machine

Code: Select all

ip route add 192.168.1.0/24  via 10.0.1.1 
ip route add 192.168.2.0/24  via 10.0.1.1

In router machine you need to enable ip forwarding and create iptables rules like

Code: Select all

iptables -A FORWARD -j ACCEPT
iptables -t nat -A POSTROUTING -j MASQUERADE

Of course you can fine tune these rules to specify subnet or input interface etc.
You have a very complicated setup,I give you that

You have
192.168.2.0/24
192.168.1.0/24
10.0.1.0/24
10.0.0.0/16
What is going on here.
Maybe a diagram would help.

frank0x01 · Post by **frank0x01** » Thu Apr 06, 2023 6:53 pm

Thank you for your reply, I will try your suggestions and report back.

I can see that the setup is a bit confusing, but that is how I have set it up at the moment for experimenting with the network configuration. It will not be the final configuration:

https://imgur.com/a/VjEviXY

I've thrown my full google fu at it for days, but with little progress till now.

Thanks again!

frank0x01 · Post by **frank0x01** » Thu Apr 06, 2023 7:28 pm

YES! Finally success. With a working configuration I can now proceed with setting up the bpi-r2.

For completeness the current configuration:

bpi-r2 router

Code: Select all

ip a:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 12:ac:4f:48:aa:0a brd ff:ff:ff:ff:ff:ff
    inet 169.254.200.153/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.255.255/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::10ac:4fff:fe48:aa0a/64 scope link
       valid_lft forever preferred_lft forever
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:b0:fc:04:a1:73 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global wan
       valid_lft forever preferred_lft forever
    inet6 2a02:a448:7890:1:40b0:fcff:fe04:a173/64 scope global dynamic mngtmpaddr
       valid_lft 258802sec preferred_lft 172402sec
    inet6 fe80::40b0:fcff:fe04:a173/64 scope link
       valid_lft forever preferred_lft forever
5: lan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:c3:0b:dd:07:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global lan0
       valid_lft forever preferred_lft forever
    inet6 2a02:a448:7890:100:cc3:bff:fedd:784/64 scope global dynamic mngtmpaddr
       valid_lft 255599sec preferred_lft 169199sec
    inet6 fe80::cc3:bff:fedd:784/64 scope link
       valid_lft forever preferred_lft forever
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 12:ac:4f:48:aa:0a brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global lan1
       valid_lft forever preferred_lft forever
    inet6 fe80::10ac:4fff:fe48:aa0a/64 scope link
       valid_lft forever preferred_lft forever
7: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 12:ac:4f:48:aa:0a brd ff:ff:ff:ff:ff:ff
8: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 12:ac:4f:48:aa:0a brd ff:ff:ff:ff:ff:ff
9: br0: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 66:d4:30:f0:ea:d3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.1/32 brd 10.0.10.255 scope global br0
       valid_lft forever preferred_lft forever

ip r:
default via 192.168.2.254 dev wan metric 4
10.0.1.0/24 dev lan1 proto kernel scope link src 10.0.1.1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.200.153
192.168.1.0/24 dev lan0 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev wan proto kernel scope link src 192.168.2.1

lan0:
https://imgur.com/a/A0lwUYA

lan1

Code: Select all

ip a:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b2:c9:97:0b:f8:00 brd ff:ff:ff:ff:ff:ff
    inet 169.254.172.112/16 brd 169.254.255.255 scope global noprefixroute dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::e5dd:4e36:2a24:d268/64 scope link
       valid_lft forever preferred_lft forever
3: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:13:3b:0f:37:be brd ff:ff:ff:ff:ff:ff
4: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:13:3b:0f:37:bf brd ff:ff:ff:ff:ff:ff
5: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 90:2b:34:6d:0e:f6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.23/16 brd 10.0.1.255 scope global dynamic noprefixroute enp7s0
       valid_lft 216107sec preferred_lft 188924sec
    inet6 fe80::4605:614b:e24c:f634/64 scope link
       valid_lft forever preferred_lft forever
6: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:c3:55:43:88 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

ip r:
default via 10.0.1.1 dev enp7s0 proto dhcp src 10.0.1.23 metric 1005
10.0.0.0/16 dev enp7s0 proto dhcp scope link src 10.0.1.23 metric 1005
169.254.0.0/16 dev dummy0 scope link src 169.254.172.112 metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.1.0/24 via 10.0.1.1 dev enp7s0
192.168.2.0/24 via 10.0.1.1 dev enp7s0

With a valid network configuration I can study it and make changes as I see fit. Thanks!

Post by **NeddySeagoon** » Thu Apr 06, 2023 7:40 pm

frank0x01,

Lets take a stop back and understand a little about routing.
To keep it simple, each interface will only have a single IP.

A system with only one real interface (ignoring lo:, sit0: and friends will have a routing table that looks like

Code: Select all

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         pi_router       0.0.0.0         UG    2      0        0 eth0
loopback        localhost       255.0.0.0       UG    0      0        0 lo
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

Routes are tested from the bottom of the routing table going up. The packet is dispatched by the first matching rule
Packets for the 192.168.100.0/24 subnet don't need any routing - they are just sent out of eth0.
Lets ignore loopback.
All other packets are sent to the gateway called Pi_router

Code: Select all

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.100.252 0.0.0.0         UG    2      0        0 eth0
...

which is in the 192.168.100.0/24 subnet.

The entire internet works like this, a system gets a packet, it either routes it or forwards it to its default route and decrements the (TTL) Time To Live.
When the TTL reaches zero, an error response it generated.

With two real interfaces in a system, life gets more complex. Having more than one default route does not make sense at the one nearest the bottom of the routing table will always be used.
If you have two internet connections then the route with the lowest Metric will be used, but lets not dwell on that right now.

When I plug another network card in and look at the routing table, it might look like

Code: Select all

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         pi_router       0.0.0.0         UG    2      0        0 eth0
loopback        localhost       255.0.0.0       UG    0      0        0 lo
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.54.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1

(its faked for this post)

Starting at the bottom everything for 192.168.54.0/24 goes out of eth1 ... nothing else is changed.

However, the system is not yet a router as packets are not forwarded between eth0 and eth1.
That requires kernel support and forwarding to be turned on.
forwarding does just that. Packets coming in on 192.168.54.0/24 destined for 192.168.100.0/24 will be forwarded as it. Its like one big network that uses two subnets.
Other systems it the network will need IP addresses in both subnets to reach both subnets.
That's not wrong, it works for a small number of systems.

To make it more generally useful, add Network Address Translation (NAT)
Now anything arriving on 192.168.54.0/24 will be mad to look as if it came from the NAT systems 192.168.100.0/24 before its sent on its way.
The reverse process will be performed for replies.

Practice that by hand. It won't work to start with but when you have done it for two interfaces and understand it, its easy to expand to more.

You can add in dhcpd, to serve addresses on the internal subnets.

Oh ...

Code: Select all

10.0.1.0/24
10.0.0.0/16

Don't have overlapping subnets like that or is that a typo?

frank0x01 · Post by **frank0x01** » Thu Apr 06, 2023 8:20 pm

NeddySeagoon

Although I have been enjoying linux for more than 2 decades I have always skimped over the network stuff. With every issue my strategy was just google, trail & error until resolved. In this case this method failed me as there is just too many subjects about linux networking that it is difficult to really grasp the gist of it. Thank you for your in-depth yet simple explanation.

PS. the 10.0.0.0/16 was a route on the Linksys router to give access from the linksys subnet to the network on the bpi-r2 router. It worked somewhat, but part of the trial and error.