Site to Site communication issues...

[rudepeople]

I'm trying to setup a site to site vpn with openvpn at both ends. This has been done before and on the surface it seems straight forward, but I guess I'm trying to do things a little weird. My company has a cluster of disparate sites with a range of network products making a uniform vpn solution rather difficult and we struggle with outages ALL the dang time. the biggest problem I'm facing is the routers. some are sonicwall, some ubiquiti, and some are cisco. None of them can have a stable s2s vpn with all the others simultaneously. the plan is to install openvpn servers at each site, BEHIND the routers and just link the sites via tunnels inside the local LANs with a set of static routes pointing back to the VPN servers for LAN2LAN communication.

right now, I'm working on one link as a proof of concept.

the layout is thus (fake IPs are fake):

Site 1

Code: Select all

clients 
- LAN: 192.168.25.255/24
vpnserver1 
- LAN: 192.168.21.10/24
EdgeRouter1
- LAN: 192.168.21.1/24
-WAN: 11.0.0.1/32

Site 2

Code: Select all

clients
-LAN: 192.168.0.255/24
vpnserver2
-LAN: 192.168.0.12/24
SonicWall2
-LAN: 192.168.0.1/24
-WAN: 12.0.0.1/32

So right off, you'll notice site 1 has vlans that will obfuscate things a bit. the VPN server and router are in vlan 21 while the clients are in vlan 25. not sure if this is a problem, but it's something to make note of.
Next off, the vpn server at site 1 is running two instances of openvpn. One for remote connections from remote users (it's a generic setup) and a second running for site 2 site.

Now for the setup, I've used a script from this site to both install and configure openvpn on server1. it's a good script and I've used it at numerous sites to get things up and running quickly. plus it lets me setup new users quick, fast, and in a hurry.

I used said script to generate a new client config for a client called "s2sclient". Then I copied the original server.conf file to a new config file called s2sserver.conf and changed it's udp port and IP config (using 1195 and 10.1.0.0 instead of 10.8.0.0) so it wouldn't conflict with openvpn@server so now I have openvpn@server and openvpn@s2sserver.

Next I updated /etc/iptables/add-openvpn-rules.sh thusly:

Code: Select all

#!/bin/sh
#tun0 rules
iptables -t nat -I POSTROUTING 1 -s 10.8.0.0/24 -o ens192 -j MASQUERADE
iptables -I INPUT 1 -i tun0 -j ACCEPT
iptables -I FORWARD 1 -i ens192 -o tun0 -j ACCEPT
iptables -I FORWARD 1 -i tun0 -o ens192 -j ACCEPT
iptables -I INPUT 1 -i ens192 -p udp --dport 1194 -j ACCEPT
#s2s tun1 rules
iptables -t nat -I POSTROUTING 1 -s 10.1.0.0/24 -o ens192 -j MASQUERADE
iptables -I INPUT 1 -i tun1 -j ACCEPT
iptables -I FORWARD 1 -i ens192 -o tun1 -j ACCEPT
iptables -I FORWARD 1 -i tun1 -o ens192 -j ACCEPT
iptables -I INPUT 1 -i ens192 -p udp --dport 1195 -j ACCEPT

Next I rebooted the server and checked that my normal remote clients can still connect... Good so far!

Now, the vpn server at site 2 is running windows 2k12 so I just installed the openvpn community client for win7 (as suggested) and imported the s2sclient.ovpn. Connect... and the connection is successful, I have a basic vpn from the site 2 vpn server to site 1 and I have a stable connection... sort of... I am also having the issue described here and here and here... yeah... it's a pretty common problem. but it seems to be a problem with the OS and I don't care to address it here. Running ipconfig /renew "tun1" gets me around it.

At this point I can ping and connect to most network devices on the site1 network from the site2 vpn server and vice versa. so now I need to get routing up and running. For this I adapted the guide here. Basically all I did was run the following commands:

Code: Select all

# mkdir /etc/openvpn/s2s1ccd
# touch /etc/openvpn/s2s1ccd/client
# echo “iroute 192.168.0.12 255.255.255.0” > /etc/openvpn/s2s1ccd/client

...add this to the s2sserver.conf:

View OriginalServer Config

#S2S routing
push "route 192.168.21.0 255.255.255.0"
push "route 192.168.22.0 255.255.255.0"
#push "route 192.168.23.0 255.255.255.0"
#push "route 192.168.24.0 255.255.255.0"
#push "route 192.168.25.0 255.255.255.0"
#push "route 192.168.26.0 255.255.255.0"
client-config-dir s2s1ccd
route 192.168.0.12 255.255.255.0
#/S2S routing

then restart the s2sserver and client...

NOTE: I have 4 of the 6 routes commented out because the tunnels for vlans 21 and 22 don't work on the existing VPN anyway and therefore won't cause routing conflicts, while vlans 23-26 DO work and might gum things up if suddenly a second route for each appears. Once I have 21 and 22 working, I'll enable the other 4 and bring down the existing crap-tastic vpn.

From here, things appear to be setup correctly. from both the client and server, I am still able to ping and connect to network resources on the other side of the vpn without incident. however, this is where things stop making progress...

on both ends I have a test client. I assume that if I change my test client's default gateway to point to the VPN server at either location, I should be able to see remote resources... right? ...because I can't. while each vpn server is able to connect to the other and browse network resources, none of the clients behind them can. I even went so far as to disable the firewall on the win2k12 server to make sure it wasn't a firewall issue, and as you can see the firewall (iptables) on the linux server is set to allow all traffic through those tunnels... so what am I missing?

on the win2k12 side I checked the logs to be sure the routes are pushing and I see the following:

Code: Select all

Mon Sep 23 09:59:59 2019 us=493445 PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 192.168.24.18,dhcp-option DNS 192.168.0.12,redirect-gateway def1 bypass-dhcp,route 192.168.21.0 255.255.255.0,route 192.168.22.0 255.255.255.0,route-gateway 10.1.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.1.0.2 255.255.255.0,peer-id 1,cipher AES-128-GCM'

As noted above, I can see the two desired routes for 192.168.21.0 and 192.168.22.0... so what gives?

I thought it might be the ccd, but if that were the case then I'd expect the clients at site1 to not see site2 resources, but then site2 clients should still be able to see site1 resources... right?

[rudepeople]

UPDATE:

I have eliminated windows and it's pesky network stack from the equation and installed openvpn on another Debian server. I then copied the s2sclient.ovpn to /etc/openvpn/s2sclient.conf and rebooted.

I have made NO other changes, save for setting up iptables with the following:

Code: Select all

#!/bin/sh
#tun0 rules
iptables -t nat -I POSTROUTING 1 -s 10.8.0.0/24 -o ens192 -j MASQUERADE
iptables -I INPUT 1 -i tun0 -j ACCEPT
iptables -I FORWARD 1 -i ens192 -o tun0 -j ACCEPT
iptables -I FORWARD 1 -i tun0 -o ens192 -j ACCEPT
iptables -I INPUT 1 -i ens192 -p udp --dport 1194 -j ACCEPT

and here are the server and client configs...

View Originals2sserver.conf

port 1195
proto udp
dev tun
user nobody
group nogroup
persist-key
persist-tun
keepalive 10 120
topology subnet
server 10.1.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 192.168.24.18"
push "dhcp-option DNS 192.168.0.12"
push "redirect-gateway def1 bypass-dhcp"
#BEGIN S2S routing info
push "route 192.168.21.0 255.255.255.0"
push "route 192.168.22.0 255.255.255.0"
#push "route 192.168.23.0 255.255.255.0"
#push "route 192.168.24.0 255.255.255.0"
#push "route 192.168.25.0 255.255.255.0"
#push "route 192.168.26.0 255.255.255.0"
client-config-dir s2s1ccd
route 192.168.0.12 255.255.255.0
##END S2S routing info
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key 0
crl-verify crl.pem
ca ca.crt
cert server[*redacted*].crt
key server[*redacted*].key
auth SHA256
cipher AES-128-GCM
ncp-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
status /var/log/openvpn/status.log
verb 3

View Originals2sclient

client
proto udp
remote *redacted* 1195
dev tun
resolv-retry infinite
nobind
persist-key
persist-tun
#dev-type tun
#dev-node tun0
remote-cert-tls server
verify-x509-name server[*redacted*] name
auth SHA256
auth-nocache
cipher AES-128-GCM
tls-client
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
#setenv opt block-outside-dns # Prevent Windows 10 DNS leak
verb 4
<ca>
*redacted*
</ca>
<cert>
*redacted*
</cert>
<key>
*redacted*
</key>
<tls-crypt>
*redacted*
</tls-crypt>

I'm seeing the same activity. from BOTH servers, I can ping and connect to all network resources on BOTH sides of the VPN. I can NOT connect through them from any of our network clients...

EDIT: heh... the oconf tag stripped out the "REDACTED" lines I had entered to replace the crypto segments... I guess I didn't have to remove it myself (still gonna do it ITF)!

[TinCanTech]

You read the official openvpn howto as well, right .. ?

[rudepeople]

I did, but the guide is significantly obfuscated to the point I'm not sure what the heck it's talking about half the time.

care to elaborate as to what portion of the guide you think I should be reading specifically?

yeah... sarcasm asside, I'm pretty sure I'm facing a bridging issue here. I'll get a tap and bridge built and go from there.

I'll post back my results.

[EDIT]:
Scratch ALL of that. I found my issue. I am dumb.

as suggested by TinCanTech, I found the fix here.

I was pushing a route to the vpn client and not it's network in both the ccd and in the server config. Also, I had named the ccd FOLDER after the connecting client instead of the client file itself so the vpn was unable to route properly. I also added client-to-client because eventually I will have several of these running and we'd like to have everyone able to see everyone else.

here is the correct configs for anyone interested:

/etc/iptables/add-openvpn-rules.sh

Code: Select all

#!/bin/sh
#tun0 rules
iptables -t nat -I POSTROUTING 1 -s 10.8.0.0/24 -o eth0 -j MASQUERADE
iptables -I INPUT 1 -i tun0 -j ACCEPT
iptables -I FORWARD 1 -i eth0 -o tun0 -j ACCEPT
iptables -I FORWARD 1 -i tun0 -o eth0  -j ACCEPT
iptables -I INPUT 1 -i eth0 -p udp --dport 1194 -j ACCEPT
#site-to-site - tun1 rules
iptables -t nat -I POSTROUTING 1 -s 10.1.0.0/24 -o eth0 -j MASQUERADE
iptables -I INPUT 1 -i tun1 -j ACCEPT
iptables -I FORWARD 1 -i eth0 -o tun1 -j ACCEPT
iptables -I FORWARD 1 -i tun1 -o eth0 -j ACCEPT
iptables -I INPUT 1 -i eth0 -p udp --dport 1195 -j ACCEPT

/etc/openvpn/ccd/s2sclient:

Code: Select all

iroute 192.168.0.0 255.255.255.0

/etc/openvpn/s2sserver.conf

View Originals2sserver.conf

port 1195
proto udp
dev tun
user nobody
group nogroup
persist-key
persist-tun
keepalive 10 120
topology subnet
server 10.1.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 192.168.24.18"
push "dhcp-option DNS 192.168.0.12"
push "redirect-gateway def1 bypass-dhcp"
#BEGIN S2S routing info
push "route 192.168.21.0 255.255.255.0"
push "route 192.168.22.0 255.255.255.0"
#push "route 192.168.23.0 255.255.255.0"
#push "route 192.168.24.0 255.255.255.0"
#push "route 192.168.25.0 255.255.255.0"
#push "route 192.168.26.0 255.255.255.0"
client-config-dir ccd
route 192.168.0.0 255.255.255.0
client-to-client
push "route 192.168.0.0 255.255.255.0"
##END S2S routing info
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key 0
crl-verify crl.pem
ca ca.crt
cert server_[redacted].crt
key server_[redacted].key
auth SHA256
cipher AES-128-GCM
ncp-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
status /var/log/openvpn/status.log
verb 3

/etc/openvpn/s2sclient.conf

View Originals2sclient.conf

client
proto udp
remote [redacted] 1195
dev tun
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
verify-x509-name server_[redacted] name
auth SHA256
auth-nocache
cipher AES-128-GCM
tls-client
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
verb 4
<ca></ca>
<cert></cert>
<key></key>
<tls-crypt></tls-crypt>

Now, to repeat all of this in bridge mode... I need multicast to work for provisioning voip phones and I doubt that will work without a bridge.

[TinCanTech]

You will probably want to read this:
https://community.openvpn.net/openvpn/w ... versubnet.

Edit: You posted your update moments before I posted my reply, sorry about that.

I still recommend you read the official Howto carefully

[rudepeople]

Edit: You posted your update moments before I posted my reply, sorry about that.

I still recommend you read the official Howto carefully

Annnd I spoke too soon!

I have gone over the howto quite thoroughly, and yes, I did miss some things... but I'm back at a fail.

during all of this, I had an old and broken VPN configured between the routers on either side. two of the tunnels are dead and I have long since given up bringing them online... As stated, I'm dumb so I forgot to delete the tunnels before starting this.

Oddly enough, setting up the above config allowed ICMP to work over those tunnels! ...but nothing else.

figured it out when I disabled the old VPN entirely and tried to use the new one... yeah... all dead.

With the above config, I am able to connect the two servers (client/server config) and from each, I can ping and connect to resources on the other side. but I still can't communicate from a client over the vpn.

I have since killed the old VPN to eliminate cross contamination, set a new static route on the router to directed all traffic going to 192.168.21.0/24 (pointing to 192.168.0.11) and tried a tracert from one of clients.

heres what I get:

Code: Select all

c:\nyxcrap\Shares\Applications>tracert 192.168.21.1

Tracing route to 192.168.21.1 over a maximum of 30 hops

  1     1 ms     1 ms     2 ms  192.168.0.11
  2    48 ms    42 ms    38 ms  10.1.0.1
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.

10.1.0.1 is the vpn server at the other site so traffic is going that far. likewise, a tracert to 192.168.21.11 (the remote VPN servers IP) works like a champ, and I can even ssh into it from here! ...so like I said, I'm *almost* there... but somethings still mucked up somewhere!

Like I said, I have gone over the howto several more times and I'm not seeing anything amiss...

[rudepeople]

Maybe I'm barking up the wrong tree, but how do I check to confirm the ccd directory and client file are actually being called? I tried checking syslog and /var/log/openvpn/status.log and even tried looking for anything in systemctl but I cant really tell if the ccd is being loaded.

[rudepeople]

something else interesting, this is the contents of /var/log/openvpn/status.log on the vpn server

Code: Select all

OpenVPN CLIENT LIST
Updated,Tue Sep 24 13:54:43 2019
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
s2sswift,[redacted],29442,37965,Tue Sep 24 13:50:26 2019
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
192.168.0.89C,s2sclient,[redacted],Tue Sep 24 13:54:40 2019
192.168.0.0/24,s2sclient,[redacted],Tue Sep 24 13:50:26 2019
10.1.0.2,s2sclient,[redacted],Tue Sep 24 13:54:33 2019
GLOBAL STATS
Max bcast/mcast queue length,1
END

My client machine (not the remote server, but a terminal downstream from it) is IPd 192.168.0.89... the remote server is 192.168.0.11... so this entry has me a little intrigued. the entry only appears when I ping the vpn server.

[rudepeople]

ERG! this is so frustrating!!! I'm actually connected via ssh to the vpn server from a client connecting THROUGH the remote server (client-->vpn-remote-server-->router-->internet<--router<--vpn-server), but I can't see anything past it!!!

[rudepeople]

UPDATE: I got one side working.
From the local network I set one of the clients default gateway as the vpn server's local IP and it just started working from that side pointed at the remote server. I'm pretty sure the static route I set on the router is broken, but I'm not willing to address it until this works both ways...

So THAT's working, however I'm still not able to connect to local resources with remote terminals, even when I set the remote vpn server as their default gateway.

Now it's starting to look like a ccd issue, but I'm not sure what to change... maybe the client file in the ccd directory is named improperly?

I keep calling the remote server "s2sclient" but when I created it, I used the name "s2sswift". as such I created /etc/openvpn/ccd/s2sswift and added the iroute line to that... Is that right? or should I have gone with /etc/openvpn/ccd/client?
Also in the s2sserver.conf file I used the line "client-config-dir ccd" should that be "client-config-dir /etc/openvpn/ccd/" or "client-config-dir /etc/openvpn/ccd/s2sswift" instead?

[rudepeople]

I hate to necro-bump, but my issue remains... I get that no one has any solutions, but maybe someone can point me toward another forum to ask?

[TinCanTech]

I hate to necro-bump, but my issue remains

Sometimes it has to be done.

Maybe I'm barking up the wrong tree, but how do I check to confirm the ccd directory and client file are actually being called?

You can check the server log or add --ccd-exclusive (See the manual) to the server config.

[rudepeople]

You can check the server log or add --ccd-exclusive (See the manual) to the server config.
[/quote]

Well, after quite a bit of tiral and error I tried running openvpn alone (stopped the service) and just watched... turns out it WAS wrong.

kept getting this:

Code: Select all

 OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd//s2sswift

so I removed the "/" at the end of the ccd line and it seems to be connecting a little better...

as of now, I can ping the openvpn server AND my test workstation from the other side of the vpn... but nothing else... the server is at 192.168.21.11 and the test station is at 192.168.21.254... I have servers at 192.168.21.5, 192.168.21.6, and 192.168.21.10. I also have a router at 192.168.21.1 and switches at 192.168.21.2 and 192.168.21.3. nothing else can be pinged from the remote site. HOWEVER, I can ping any of the 3 servers and 5 workstations on the remote side from all of the above!

I have no clue where to go from here... it's almost like theres something filtering the packets somewhere, but like I said before, I can ping everything on both sides from the client machine on the remote side, just nothing that tries to use it as a gateway!

[rudepeople]

Alright, I have the pattern.

From the remote side, I can see everything on the server side. From the server side, I can only see devices that are using the vpn server itself as their default gateway.