So. . . I have a weird, complicated setup and I've been messing with it a ton over the last several days. I suspect my problem has something to do with either a routing config that needs to get pushed or something firewalling something else somewhere. I also suspect it might be a netmask issue?
Here's my problem.
I have an existing network that lives on a 172.28.254.* At the gateway of this network is a Watchguard firewall that lives at 172.28.252.1. It's running Fireware's implementation of OpenVPN, which is allowing close to 50-60 remote workers to currently work from home and access lots of OnPrem resources, including our Domain Controller and a bunch of linux VMs running in Xen for various network services. Those resources live on a 172.27.252.* subnet. Things are currently set up in the Watchguard such that addresses on the 172.28.254.* subnet can ping (and route to) VPN clients on the 172.27.252.* subnet. All of that is working fine.
I've added a second OpenVPN server on the network (currently running on a bare metal box running Ubuntu server). The purpose of this secondVPN is to add cloud resources to the network and have them behave as local resources. This VPN is set up as a network bridge and is currently mostly working.
Clients receive addresses in the 172.28.254.* range. They even get them from our DHCP server, living on the Watchguard (or they seem to, since they're getting addresses like 172.28.254.37, but the OpenVPN address pool is 172.28.254.80-89).
Ips physically on the network (or even bridged to it in the case of Xen virtual machines) can talk to addresses on the second VPN just fine. For example, 172.28.254.60 is a Xen VM running CentOS hosted on our on prem server. It can talk back and forth normally with 172.28.254.34 which is a VM running Ubuntu Server running on my local machine at home. The ips on the second VPN can also talk to the Watchguard at 172.28.252.1 with no problems (I can ping it and load it's config page from a VPN client).
So we have what we'll call
The OnPrem network living 172.28.254.* GW is the Watchguard at 172.28.252.1.
We have VPN A living at 172.27.252.* VPN server is the Watchguard.
And VPN B sharing address space with 172.28.254.* VPN server is an Ubuntu server living at 172.28.254.22.
The problem comes when a client on VPN A attempts to talk to a client on VPN B (or vice versa). This just doesn't work no matter what I do.
I have spent days playing with routes. My current server side routing config says:
server-bridge 172.28.252.1 255.255.252.0 172.28.254.80 172.28.254.89
push "route 172.28.254.0 255.255.252.0"
client-to-client
route 172.28.254.0 255.255.252.0
I've had routes in there for 172.27.252.0, but it seems to make no difference. I have also tried pushing gateways in the routes, so you know, this (and a number of less likely variants) already didn't work:
server-bridge 172.28.252.1 255.255.252.0 172.28.254.80 172.28.254.89
push "route 172.28.254.0 255.255.252.0"
push "route 172.27.252.0 255.255.255.0 172.28.252.1"
route 172.28.254.0 255.255.252.0
route 172.27.252.0 255.255.255.0 172.28.252.1"
Here's the routing table for an onprem machine (actually the VPN2 server):
default _gateway 0.0.0.0 UG 0 0 0 br0
172.27.252.0 _gateway 255.255.255.0 UG 0 0 0 br0
172.28.252.0 0.0.0.0 255.255.252.0 U 0 0 0 br0
Here's the routing table for a VPN A client (as it pertains to the tun0 adapter):
172.27.252.0 0.0.0.0 255.255.255.0 U 50 0 0 tun0
172.28.252.0 _gateway 255.255.252.0 UG 50 0 0 tun0
Here's the routing table of a VPN B client (as it pertains to the tap0 adapter)
172.28.252.0 0.0.0.0 255.255.252.0 U 0 0 0 tap0
_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 tap0
Here's the part where I suspect it may be a netmask issue:
I noticed that, even when I was pushing the 172.27.252.0 option in server.conf that the VPN B clients didn't have it in their routing table. So I tried to add it manually (using the same route an on prem machine would have) with
route add 172.27.252.0 gw 172.28.252.1 netmask 255.255.255.0
I would get the error "route: netmask 000000ff doesn't make sense with host route"
Which makes sense. . . but that's the netmask that the on prem machines show in their routing table, that they get from the DHCP server. I ALSO see in the openVPN logs (even without the 172.27.252.0 route being pushed) that it's trying to add a route and failing (status 2).
Does anyone have any ideas of where I might go from here? Thanks very much.