routes to remote subnets are incorrect

Ace8Track

I am running OpenVPN server on Ubuntu server.

I have several client tunnels up, and I can ping all the tunnel endpoints from the server.

I have ccd files with the remote subnets defined, and the route set in the server.conf file.

This worked just fine before, but now all the routes installed on the OS to remote subnets point to 10.8.0.2 instead the their correct endpoint (10.8.0.4 or 10.8.0.6 and so on).

Anyone else run into this or know what the problem may be?

Thanks,
Arrie

TinCanTech

Ace8Track wrote but now all the routes installed on the OS to remote subnets point to 10.8.0.2

That is correct.

Ace8Track

10.8.0.2 is the tunnel endpoint of one of the many clients. Yet all the remotes subnets for all the clients are pointing to there. This isn't correct.

For instance we have a remote client called RNO4. There a subnet at RNO4 10.18.1.0/24. The tunnel endpoint for RNO4 is 10.8.0.5.

When I run "ip route", the route installed is "10.8.1.0/24 via 10.8.0.2 dev tun0". This is wrong, as 10.8.0.2 is the tunnel endpoint for RNO1, a totally different site.

If I manually change the route "10.8.1.0/24 via 10.8.0.5 dev tun0", it works just fine. But this route get reset back to 10.8.0.2 (RNO1) every time the service is restarted.

TinCanTech

That is the correct route for the server.

Ace8Track

No it is not the correct route for the server.

There are a lot of remote subnets that go to different clients. The server needs routes to these remote subnets that point to the correct client endpoint. If I manually change the routes, everything works great, but the openvpn process should install these routes correct automatically, and has in the past.

Trying to figure out why its not now.

TinCanTech

That is the correct route for the server.

I suggest you start here:
https://forums.openvpn.net/viewtopic.php?f=30&t=22603

Ace8Track

Thanks for your insight. Great explanation. Very helpful.

Ace8Track

I'm tying to understand why. Here is route table when not working:

arrie@oob-openvpn:/etc/openvpn$ ip route
default via 10.64.0.1 dev ens4 proto dhcp src 10.64.0.6 metric 100 
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1 
10.64.0.1 dev ens4 proto dhcp scope link src 10.64.0.6 metric 100 
10.124.9.0/24 via 10.8.0.2 dev tun0 
10.124.126.0/24 via 10.8.0.2 dev tun0

The 10.124 subnets are pointing to a client endpoint that are the wrong site for these subnets. See below:

arrie@oob-openvpn:~$ sudo cat /var/log/openvpn/status.log
OpenVPN CLIENT LIST
Updated,Wed Feb 24 18:46:19 2021
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
chi5,XX.XX.228.202:62497,4392,4386,Wed Feb 24 18:44:52 2021
sac1,XX.XX.253.28:62120,4276,4386,Wed Feb 24 18:44:50 2021
nyc17,XX.XX.253.8:61442,4160,4154,Wed Feb 24 18:45:13 2021
chi3,XX.XX.253.36:61102,4160,4154,Wed Feb 24 18:45:06 2021
sac2,XX.XX.253.39:52549,5795,3806,Wed Feb 24 18:45:41 2021
lax16,XX.XX.253.45:61544,3824,3638,Wed Feb 24 18:46:13 2021
wdc8,XX.XX.253.34:54425,3812,3806,Wed Feb 24 18:45:46 2021
jax2,XX.XX.148.8:53083,3811,3806,Wed Feb 24 18:45:39 2021
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
10.8.0.9,chi3,XX.XX.253.36:61102,Wed Feb 24 18:45:08 2021
10.8.0.5,jax2,XX.XX.148.8:53083,Wed Feb 24 18:45:40 2021
10.8.0.2,lax16,XX.XX.253.45:61544,Wed Feb 24 18:46:18 2021
10.8.0.8,nyc17,XX.XX.253.8:61442,Wed Feb 24 18:45:13 2021
10.8.0.6,sac2,XX.XX.253.39:52549,Wed Feb 24 18:46:16 2021
10.124.9.0/24,chi3,XX.XX.253.36:61102,Wed Feb 24 18:45:08 2021
10.8.0.3,sac1,XX.XX.253.28:62120,Wed Feb 24 18:44:51 2021
10.8.0.4,chi5,XX.XX.228.202:62497,Wed Feb 24 18:44:53 2021
10.124.126.0/24,wdc8,XX.XX.253.34:54425,Wed Feb 24 18:45:47 2021
10.8.0.7,wdc8,XX.XX.253.34:54425,Wed Feb 24 18:45:47 2021
GLOBAL STATS
Max bcast/mcast queue length,8
END

You can see the 10.124 entries are correct here in the log.

If I manually change the routes in the OS's routing table, every thing works great. See below:

arrie@oob-openvpn:~$ ip route
default via 10.64.0.1 dev ens4 proto dhcp src 10.64.0.6 metric 100 
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1 
10.64.0.1 dev ens4 proto dhcp scope link src 10.64.0.6 metric 100 
10.124.9.0/24 via 10.8.0.6 dev tun0 
10.124.126.0/24 via 10.8.0.4 dev tun0 
arrie@oob-openvpn:~$ ping 10.124.9.1
PING 10.124.9.1 (10.124.9.1) 56(84) bytes of data.
64 bytes from 10.124.9.1: icmp_seq=1 ttl=254 time=471 ms
64 bytes from 10.124.9.1: icmp_seq=2 ttl=254 time=442 ms
64 bytes from 10.124.9.1: icmp_seq=3 ttl=254 time=555 ms
64 bytes from 10.124.9.1: icmp_seq=4 ttl=254 time=606 ms
64 bytes from 10.124.9.1: icmp_seq=5 ttl=254 time=458 ms
64 bytes from 10.124.9.1: icmp_seq=6 ttl=254 time=287 ms
64 bytes from 10.124.9.1: icmp_seq=7 ttl=254 time=389 ms
64 bytes from 10.124.9.1: icmp_seq=8 ttl=254 time=202 ms
--- 10.124.9.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7002ms
rtt min/avg/max/mdev = 201.629/426.290/606.324/124.270 ms
arrie@oob-openvpn:~$ ping 10.124.126.1
PING 10.124.126.1 (10.124.126.1) 56(84) bytes of data.
64 bytes from 10.124.126.1: icmp_seq=1 ttl=254 time=163 ms
64 bytes from 10.124.126.1: icmp_seq=2 ttl=254 time=161 ms
64 bytes from 10.124.126.1: icmp_seq=3 ttl=254 time=160 ms
64 bytes from 10.124.126.1: icmp_seq=4 ttl=254 time=159 ms
64 bytes from 10.124.126.1: icmp_seq=5 ttl=254 time=156 ms
--- 10.124.126.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 155.675/159.684/163.300/2.507 ms

If I'm misunderstanding I apologize. A more directed response to the solution would be very helpful. Thanks

Pippin

Please post configs and ccd files.

Remove private data before posting.

TinCanTech

Ace8Track wrote Thanks for your insight. Great explanation. Very helpful

The reason is because the kernel routing table is not the openvpn routing table.

Ace8Track

Thanks for the responses. I'll keep digging, may be an issue with something else if you're saying OpenVPN is behaving properly.

TinCanTech

When you use --server in your config, the only gateway the server can see on the VPN subnet is the tun device. From the server point of view, the tun device remote end point is always x.x.x.2 (Even with --topology subnet).
Then the --iroute (Internal routes) settings configure openvpn so that it alone knows which client owns which remote subnet.

In --topology net30, the client sees its own remote end point from ifconfig as the route gateway for anything over the VPN. In --topology subnet, the client does use the actual server VPN IP as the route gateway.

--topology subnet is highly recommended.