OpenVPN with a br0 and tun0 interface assistance needed

[mondjef]

I am trying to get a working openvpn configuration on my Gentoo based server to allow remote internet access through my home network while remotely connected to public wifi or via mobile data on a client. I have currently a working wireguard instance running directly on my router (ASUS AX88U Pro) which is running Merlin, however the udp only protocol does not work well with many of the public wifi and thus would like to use tcp (I am aware of the latency issues....but this will at most have 2 clients connected). The router also supports openvpn and can be configured to use tcp, however the issue is that I want to use port 443 to ensure in most case I will get a connection to the openvpn server but I also host web sites on this port so I plan to eventually use something like sslh to allow hosting both openvpn and web services on the same port. For now, I just want to get openvpn up and running correctly on port tcp 443 so I can then tackle the next challenge with this setup....

I have tried so many things that my head is spinning out of control, I can get a client to connect and receive an IP address but beyond that nothing seems to work and I think it is because there is no DNS working for the connected client or packets do not make it back to the client.

Network setup:

Code: Select all

Internet (Public IP)-
                     \-> Asus router (192.168.0.1)-
                                                   \-> Gentoo server (192.168.0.2)
                                                          -br0 (192.168.0.2 -eth0, eth1, eth2, eth3)
                                                          -tun0 (OpenVPN - 10.8.0.1)

On the gentoo sever I have a pihole instance (via docker) providing DNS for LAN that is set to respond to all request on br0 interface that I am wanting to have VPN clients also use for DNS.


current /etc/openvpv/openvpn.conf

Code: Select all

proto tcp-server
port 443
dev tun0
mode server
ca /etc/openvpn/cert/ca.crt
cert /etc/openvpn/cert/server_clinic_8amd.crt
key /etc/openvpn/cert/server_clinic_8amd.key
dh /etc/openvpn/cert/dh.pem
topology subnet
fast-io

server 10.8.0.0 255.255.255.0
#client-to-client
#ifconfig-pool-persist ipp.txt
#client-config-dir ccd

# update vpn clients IP routing table
push "route 192.168.0.0 255.255.255.0 vpn_gateway 500"
push "redirect-gateway def1"
push "dhcp-option DNS 192.168.0.2"

# connection
keepalive 15 60
tls-auth /etc/openvpn/cert/ta.key

tun-mtu 1500
tun-mtu-extra 32
mssfix 1200
duplicate-cn
comp-lzo
max-clients 10

# persistent device and key settings
persist-key
persist-tun

# logging
status openvpn-status.log
log /var/log/openvpn.log
log-append /var/log/openvpn.log

# notify clients on deamon restart to reconnect quickly
explicit-exit-notify 1

# run the server as system user nobody
user nobody
group nobody
verb 3

ip route

Code: Select all

default via 192.168.0.1 dev br0 metric 36 
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1 
127.0.0.0/8 via 127.0.0.1 dev lo 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-dd53b6059424 proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev br-fa7e21c91b26 proto kernel scope link src 172.19.0.1 
192.168.0.0/24 dev br0 proto kernel scope link src 192.168.0.2 

based on tutorials and googling I added the following iptables rules...

Code: Select all

-A FORWARD -i br0 -o tun0 -j ACCEPT
-A FORWARD -i tun0 -o br0 -j ACCEPT

My basic understanding of my setup is that my gentoo server needs to know how to route traffic between these 2 subnets and my router also needs to know how to route the traffic as well I think....but beyond that I am not sure what I need to do (or undo). Any help would be appreciated. Note: I also tried going the tap0 route and adding this interface to existing bridge br0 but this is not an option as some of the clients are Android phones where the client does not support tap and my understanding is that tap should not be used in 99% of use cases.

[salahx]

There are 2 things you need to do:

You need to enable ipv4 forwarding.
Because the VPN server is not the default gateway, you need to enable NAT on the server.

Without IP forwarding enabled, no routing will be done
Without NAT, packets will get out, but there won;t be a return path (additionally the default gateway may refuse to route packets that it doesn't believe can come behind it).

[mondjef]

There are 2 things you need to do:

You need to enable ipv4 forwarding.
Because the VPN server is not the default gateway, you need to enable NAT on the server.

Without IP forwarding enabled, no routing will be done
Without NAT, packets will get out, but there won;t be a return path (additionally the default gateway may refuse to route packets that it doesn't believe can come behind it).

ipv4 forwarding is enabled as in 'net.ipv4.ip_forward = 1' is set in /etc/sysctl.conf (was already as at one point this server served as my main router/firewall).

By NAT, do you mean I need to enable masquerading in iptables on that server?

[John R. Graham]

@mondjef, a few comments:

1. Try to take DNS out of the equation for the time being. Can you ping machines on your 192.168/24 subnet from a VPN client machine on your 10.8/24 VPN subnet? If not, then you have a routing problem.
2. Unless you have a specific reason, it's better to run your VPN tunnel on UDP rather than TCP. The latter causes packets to be double-wrapped in TCP headers, causing two sets of retry logic to be in play. The inner TCP protocol will do all of this for you so it's not necessary on the outer one.
3. It's cleaner to give your DNS server container its own IP address.

@salahx, I don't believe NAT is necessary in this situation (beyond what is obviously happening on the router).

- John

[mondjef]

@mondjef, a few comments:

1. Try to take DNS out of the equation for the time being. Can you ping machines on your 192.168/24 subnet from a VPN client machine on your 10.8/24 VPN subnet? If not, then you have a routing problem.
2. Unless you have a specific reason, it's better to run your VPN tunnel on UDP rather than TCP. The latter causes packets to be double-wrapped in TCP headers, causing two sets of retry logic to be in play. The inner TCP protocol will do all of this for you so it's not necessary on the outer one.
3. It's cleaner to give your DNS server container its own IP address.

@salahx, I don't believe NAT is necessary in this situation (beyond what is obviously happening on the router).

- John

The pihole is running via docker with host networking and binds and listens on the br0 interface (192.168.0.2) on that same server. I cannot ping other clients while connected as client vpn, I am pretty sure it is some routing issue...not sure if locally on the Gentoo server I am running openvpn on or the Asus router or combination there of.

Yes, I am aware of the issues that come with that of using openvpn over tcp vs udp but the whole reason of me doing this is for the use case of being able to connect via vpn on typically unblocked ports (and protocols) for use on more locked down public/corpoarte wifi networks. I already have wireguard working as my first option when it can be used.

When you say 'give your DNS server container its own IP address' do you mean its own ip on my main network or something? It has its own ip address on the docker virtual network that is routed via the bridge (192.168.0.2).

[salahx]

I think you need NAT here, Without NAT (or special routing rules) you'll be able to get out past the VPN server, but the return path will be through the default gateway, rather than through the VPN server. The default gateway will either route it to the Internet (where it'll be dropped by the next hop) or just drop it itself. Packets destined to the Internet /(assuming a full tunnel) will probably be dropped either because the router will only create NAT rules for the directly connected network. So anything it sees wiith 10.0.0.1 will likely go nowhere.

[mondjef]

On the Asus router, when I enable static routes and enter the openvpn interface address (10.8.0.1) and the bridge interface br0 (192.168.0.2) as the gateway to use this the corresponding iptables entry that gets added from doing so....

Code: Select all

-A FORWARD -s 192.168.0.0/24 -d 10.8.0.0/24 -i br0 -o br0 -j ACCEPT

I am not an iptables firewall expect but this rule makes me think that clients on 192.168.0.0/24 network would be able to reach 10.8.0.0/24 network but I don't think that is enough for 10.8.0.0/24 clients to try and reach 192.168.0.0/24 clients.

[salahx]

A static route may work for accessing your LAN from the VPN, but you still won't be able to access the Internet from the VPN because the router has to be told to masquerade the 10.8.0.0/24 network.

Also the static route that needs to be set the default gateway (Asus router) would need to be route 10.8.0.0/24 network to 192.168.0.2 via the LAN interface. You might want to use the traceroute command on the VPN server (using -i tun0 option) to verify the packets are getting to the default gateway.

Still, this is very fragile, and, for most consumer routers, it won't work at all, because the router will never NAT from an outside network. I still think its prefereable to enable NAT on the VPN server, and masquare the 10.8.0.0/24 network that way.

[mondjef]

Some success....

On the Gentoo server I added the following to iptables

Code: Select all

-A INPUT -i tun0 -j ACCEPT
-A INPUT -i br0 -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT

-A FORWARD -i tun0 -j ACCEPT
-A FORWARD -i tun0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT

-A POSTROUTING -s 10.8.0.0/24 -o br0 -j MASQUERADE

The result is that from my android phone (on mobile data) I can now connect to openvpn, I can ping LAN clients on the 192.168.0.0/24 network including the bridge interface (br0) on the same Gentoo server with an ip 192.168.0.2 and which hosts my pihole DNS service. However, DNS does not seem to work as beyond pinging ip addresses name resolution does not work. I am not sure if all the iptables rules above were necessary or if it really was just the NAT masquerade one that actually did any anything as on this box the default for INPUT and FORWARD chains were ACCEPT (did not try to remove them and test yet). Any thoughts on the DNS issue. I am still a bit confused on how things will need to work eventually and what exactly is needed with respect to iptables and routes. If sslh (or Nginx streaming) will be responsible for accepting traffic on the br0 interface (192.168.0.2) on port 443 and determining if the packet goes to Nginx (existing hosting on 443) or to redirect to openvpn (tun0 10.8.0.1) interface, I don't understand why iptables needs to be involved.

I do have the following in my openvpn.conf file for the server related to DNS and routes...

Code: Select all

push "route 192.168.0.0 255.255.255.0"
push "redirect-gateway def1"
push "dhcp-option DNS 192.168.0.2"

Edit: removing all iptables rules except for the NAT masquerade one seems to retain the same functionality...can connect, can ping all LAN hosts but DNS does not work.

[pietinger]

Some success....

On the Gentoo server I added the following to iptables

Code: Select all

[...]
[1] -A FORWARD -i tun0 -j ACCEPT
[2] -A FORWARD -i tun0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
[...]

Edit: removing all iptables rules except for the NAT masquerade one seems to retain the same functionality...can connect, can ping all LAN hosts but DNS does not work.

Yes, because if you dont configure any iptables rules ... the default is ACCEPT ... please look to these 2 rules:

[1] You allow EVERYTHING which comes from tun0 going to forward and then you try to allow something [2] coming from tun0 to ...no matter what it is ... it was already allowed with rule 1 This means you will see never a packet in rule 2 (when you do an "iptables -L -vn" which shows the number of packets).

I dont know how much you know about NAT ... maybe you want read this chapter (or maybe the entire article): https://wiki.gentoo.org/wiki/User:Pieti ... nation_NAT

At the moment I have no clear idea from your stations and your network ... but the first thing which should be done is always: Check that the routing in the network is correct (and I suggest to listen to our old experts like John ... )

[mondjef]

Some success....

On the Gentoo server I added the following to iptables

Code: Select all

[...]
[1] -A FORWARD -i tun0 -j ACCEPT
[2] -A FORWARD -i tun0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
[...]

Edit: removing all iptables rules except for the NAT masquerade one seems to retain the same functionality...can connect, can ping all LAN hosts but DNS does not work.

Yes, because if you dont configure any iptables rules ... the default is ACCEPT ... please look to these 2 rules:

[1] You allow EVERYTHING which comes from tun0 going to forward and then you try to allow something [2] coming from tun0 to ...no matter what it is ... it was already allowed with rule 1 This means you will see never a packet in rule 2 (when you do an "iptables -L -vn" which shows the number of packets).

I dont know how much you know about NAT ... maybe you want read this chapter (or maybe the entire article): https://wiki.gentoo.org/wiki/User:Pieti ... nation_NAT

At the moment I have no clear idea from your stations and your network ... but the first thing which should be done is always: Check that the routing in the network is correct (and I suggest to listen to our old experts like John ... )

Hence why I was suspecting they weren't doing anything and ultimately removed them. They were added out of sequence from various web sources...was grabbing at straws lol I know enough about NAT to be dangerous but not enough to be effective. I had ran a gentoo box many years as my home router using arnos-iptables-firewall package but ultimately upgraded my network with a more capable consumer based router in an effort to simplify my setup and dependency on this single machine.

I recreated my pihile docker container to have it accept DNS queries from all origins (should be ok as this box is behind router and I have not opened port 53). DNS now seems to be resolving for both LAN clients and VPN clients. Though it is only me that will be using the VPN I do question what I might want to lock down for extra security. I now need to figure out the sslh aspect so I can both serve OpenVPN and Nginx on port 443 at same time...

[mondjef]

Just coming back to this to post my working setup for others that might be venturing down this same path...

To recap, my goal was to put OpenVPN behind my Nginx proxy accessible from port 443 (OpenVPN on tcp 1194) to increase chances I would be able to reach my home network on networkds that block udp that prevents me from using wireguard. I accept that tcp is not the greatest for OpenVPN and that is fine as I use this a last resort should my wireguard connection not work. Reason for using Nginx to multiplex is that I have other services that run on port tcp 443 that need to co-exist on the machine. Note there other methods besides using Nginx in front of OpenVPN including sslh, HAproxy and as well having OpenVPN itself multiplex and send non-vpn traffic along to Nginx. The reason I chose to do it this way is to limit introducing another application into the mix to learn and configure and I did not want all my web traffic to go through OpenVPN.

On the Nginx side of things...for this to work Nginx must be compiled with the proper modules ssl_peread!

Code: Select all

stream {
    # Define a log format suitable for TCP/SSL preread
    log_format ssl_preread '$remote_addr - [$time_local] $protocol $status '
                           '$ssl_preread_server_name $bytes_sent $bytes_received '
                           '$session_time';

    access_log /var/log/nginx/tcp-access.log ssl_preread;

    map $ssl_preread_server_name $name {
        mynoip1.com web_backend;
        default vpn_backend;
    }

    server {
        listen 192.168.0.2:443 so_keepalive=on;
        proxy_pass $name;
        ssl_preread on;
        # Long timeouts can help with OpenVPN stability
        proxy_timeout 300s;
        proxy_connect_timeout 300s;
    }

    upstream vpn_backend {
        server 10.8.0.1:1194; # OpenVPN server listens here
    }

    upstream web_backend {
        server 192.168.0.2:8843; # Nginx http server listens here
    }

}

One thing to note with the nginx config is that unidentifiable domains by default go to the Openvpn backend, this is because OpenVPN does not send the SNI by my understanding so the nginx ssl preread does not work right because of that. I did setup a separate noip ddns specifically for use with my openvpn but because openvpn does not send the SNI and we are defaulting to the vpn backend there is really no need to do so.

openvpn server config

Code: Select all

proto tcp-server
port 1194
dev tun0
mode server
ca /etc/openvpn/cert/ca.crt
cert /etc/openvpn/cert/server_clinic_8amd.crt
key /etc/openvpn/cert/server_clinic_8amd.key
dh /etc/openvpn/cert/dh.pem
topology subnet
fast-io

server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
client-config-dir ccd

push "route 192.168.0.0 255.255.255.0"
push "redirect-gateway def1"
push "dhcp-option DNS 192.168.0.2"

# connection
keepalive 15 60
tls-auth /etc/openvpn/cert/ta.key

max-clients 3
persist-key
persist-tun

status openvpn-status.log
log /var/log/openvpn.log
log-append /var/log/openvpn.log

# notify clients on deamon restart to reconnect quickly
explicit-exit-notify 1

# run the server as system user nobody
user nobody
group nobody
verb 3

I do push my local dns server (pihole) to clients to use in the above server config which I have running in a docker container on the same machine.

openvpn client config example

Code: Select all

client
dev tun
proto tcp
remote mynoip1.com 443
persist-key
persist-tun
# key stuff below here...

The final thing I needed to do was ensure port 443 on my router that accesses the WAN is configured to be forwarded to machine running Nginx and on the machine running Nginx and Openvpn to nat (masquerade) traffic on the ip address of the tun interface (10.8.0.1)

[pietinger]

mondjef,

first of all: Thank you for your detailed report ... yes, it surely can help others ...

with a rule like:

Code: Select all

-A POSTROUTING -s 10.8.0.0/24 -o br0 -j MASQUERADE

you do SNAT (source NAT) ... while ... port forwarding you did in your router:

The final thing I needed to do was ensure port 443 on my router that accesses the WAN is configured to be forwarded [...]

is DNAT (destination NAT). If this server is not directly connected to your router (e.g. if you have a firewall between) then you must configure DNAT again (on the firewall) ...

So, what I want to say: Maybe it will help others even more, if you show us your network configuration (netifrc or networkmanager?) together with your final NAT configuration (iptables).

But in any case, thanks again for your report.

[mondjef]

mondjef,

first of all: Thank you for your detailed report ... yes, it surely can help others ...

with a rule like:

Code: Select all

-A POSTROUTING -s 10.8.0.0/24 -o br0 -j MASQUERADE

you do SNAT (source NAT) ... while ... port forwarding you did in your router:

The final thing I needed to do was ensure port 443 on my router that accesses the WAN is configured to be forwarded [...]

is DNAT (destination NAT). If this server is not directly connected to your router (e.g. if you have a firewall between) then you must configure DNAT again (on the firewall) ...

So, what I want to say: Maybe it will help others even more, if you show us your network configuration (netifrc or networkmanager?) together with your final NAT configuration (iptables).

But in any case, thanks again for your report.

The server is connected directly to my router (typical consumer grade router with firewall) with standard firewall functionality and port forwarding capabilities. On the server itself, there is no firewall in the sense of iptables rules intended to block traffic but iptables is there and there is the single rule there to masquerade traffic as indicated previously. The only other rules there are rules created by docker for the containers I run.

I do have to admit though, my setup might not be perfect as I seem to not be able to reach any device that is connected to any of the nic cards that are part of my br0 bridge even though they are perfectly accessible from any device connected directly to my network so not sure what is going on there but more than likely something missing in my iptables on that server is my guess.

netifrc

Code: Select all

config_eth0="null"
config_eth1="null"
config_eth2="null"
config_eth3="null"

bridge_br0="eth0 eth1 eth2 eth3"

# bridge static config
config_br0="192.168.0.2/24 brd 192.168.0.255"
routes_br0="default via 192.168.0.1"
dns_servers_br0="192.168.0.1"
mac_br0="static mac here"
bridge_stp_state_br0=0
bridge_forward_delay_br0=0
bridge_hello_time_br0=1000