Site to Site VPN - Issues on AS side

savestheday

Hi all!

I have setup a site to site VPN using the tutorial here

https://docs.openvpn.net/how-to-tutorialsguides/virtual-platforms/site-to-site-layer-3-routin-using-openvpn-access-server/

Both sites (client and AS) are running Ubuntu v16.

Everything except browsing from Site A (where the AS is) to Site B (the client) works. Clients at Site B are able to access servers/websites at Site A but clients at Site A are not able to access HTTP servers at Site B. However (and this is the part that has me puzzled), clients from Site A are able to ping said servers at Site B and are able to SSH into Site B servers. I have checked IPTables, IPv4 forwarding, etc. All of that checks out (I think). The advertised routes (both sites are running Ubiquiti Edge Routers) are correctly advertised to clients and traceroutes traverse the tunnel.

traceroute to 192.168.77.43 (192.168.77.43), 64 hops max, 52 byte packets
 1  192.168.0.1 (192.168.0.1)  1.677 ms  1.451 ms  1.064 ms
 2  192.168.0.161 (192.168.0.161)  1.906 ms  1.893 ms  1.786 ms
 3  172.27.232.2 (172.27.232.2)  6.656 ms  6.736 ms  6.532 ms
 4  192.168.77.43 (192.168.77.43)  7.158 ms  7.117 ms  6.516 ms

192.168.0.1 = Edgerouter at Site A
192.168.0.161 = OpenVPN AS server
172.27.232.2 = Client VPN IP
192.168.77.43 = Web server at Site B

telnet 192.168.77.43 80
Trying 192.168.77.43...
Connected to 192.168.77.43.
Escape character is '^]'.

I have a feeling I am missing something very simple! Any help would be appreciated, thank you!

Here is the client config
setenv FORWARD_COMPATIBLE 1
client
server-poll-timeout 4
nobind
remote test.org 1194 udp
remote test.org 1194 udp
remote test.org 443 tcp
remote test.org 1194 udp
remote test.org 1194 udp
remote test.org 1194 udp
remote test.org 1194 udp
remote test.org 1194 udp
dev tun
dev-type tun
ns-cert-type server
reneg-sec 604800
sndbuf 100000
rcvbuf 100000
# NOTE: LZO commands are pushed by the Access Server at connect time.
# NOTE: The below line doesn't disable LZO.
comp-lzo no
verb 3
setenv PUSH_PEER_INFO

The Johan

I would suggest you run tcpdump on the Access Server and on the clients and run ping tests to see where the traffic stops exactly, and how the packets are treated.

If pinging works in both directions, then usually everything is set up fine and it's just a matter of firewalls.

savestheday

novaflash wroteI would suggest you run tcpdump on the Access Server and on the clients and run ping tests to see where the traffic stops exactly, and how the packets are treated.

If pinging works in both directions, then usually everything is set up fine and it's just a matter of firewalls.

Pinging definitely works. I'll check out TCPDump and see what's going on. Thank you!

savestheday

Ok update,

pinging 192.168.77.43 or any other boxes from the OpenVPN AS server is *NOT* working. (I think this is a big tell?)
pinging 192.168.77.83 (the OpenVPN client at Site B) *IS* working.
I can ping 192.168.77.43 from other clients on site A which is weird.

Attached is the output of my IPTables

sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 20 packets, 1647 bytes)
 pkts bytes target     prot opt in     out     source               destination         
14022   12M AS0_ACCEPT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 AS0_ACCEPT  all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 AS0_IN_PRE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000000/0x2000000
    4   313 AS0_ACCEPT  udp  --  *      *       0.0.0.0/0            192.168.0.161        state NEW udp dpt:1194
    0     0 AS0_ACCEPT  tcp  --  *      *       0.0.0.0/0            192.168.0.161        state NEW tcp dpt:443
    0     0 AS0_WEBACCEPT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
   28  1680 AS0_WEBACCEPT  tcp  --  *      *       0.0.0.0/0            192.168.0.161        state NEW tcp dpt:943
  254 19785 ufw-before-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  254 19785 ufw-before-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  254 19785 ufw-after-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  243 19063 ufw-after-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  243 19063 ufw-reject-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  243 19063 ufw-track-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3105 1392K AS0_ACCEPT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
  926  173K AS0_IN_PRE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000000/0x2000000
  142  8224 AS0_OUT_S2C  all  --  *      as0t+   0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-before-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-before-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-after-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-after-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-reject-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ufw-track-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  as0t0  enp2s1  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  enp2s1 as0t0   0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  as0t1  enp2s1  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  enp2s1 as0t1   0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 26 packets, 3865 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   44  3312 AS0_OUT_LOCAL  all  --  *      as0t+   0.0.0.0/0            0.0.0.0/0           
  868  150K ufw-before-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  868  150K ufw-before-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  846  147K ufw-after-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  846  147K ufw-after-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  846  147K ufw-reject-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  846  147K ufw-track-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_ACCEPT (5 references)
 pkts bytes target     prot opt in     out     source               destination         
17131   13M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_IN (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            172.27.224.1        
  887  170K AS0_IN_POST  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_IN_NAT (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000000
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_IN_POST (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  887  170K ACCEPT     all  --  *      *       0.0.0.0/0            192.168.0.0/24      
    0     0 AS0_OUT    all  --  *      as0t+   0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_IN_PRE (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 AS0_IN     all  --  *      *       0.0.0.0/0            169.254.0.0/16      
  887  170K AS0_IN     all  --  *      *       0.0.0.0/0            192.168.0.0/16      
    0     0 AS0_IN     all  --  *      *       0.0.0.0/0            172.16.0.0/12       
    0     0 AS0_IN     all  --  *      *       0.0.0.0/0            10.0.0.0/8          
   39  2988 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_IN_ROUTE (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000000
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_OUT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0            all  --  *      *       0.0.0.0/0            0.0.0.0             
    0     0 AS0_U_SITEC_OUT  all  --  *      *       0.0.0.0/0            10.10.20.0/24       
    0     0 AS0_U_SITEB_OUT  all  --  *      *       0.0.0.0/0            172.27.232.2        
  142  8224 AS0_U_SITEB_OUT  all  --  *      *       0.0.0.0/0            192.168.77.0/24     
    0     0 AS0_OUT_POST  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_OUT_LOCAL (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 5
   44  3312 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_OUT_POST (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000000/0x2000000
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_OUT_S2C (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  142  8224 AS0_OUT    all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_U_SITEC_OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       192.168.0.0/24       0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       10.10.20.0/24        0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       192.168.77.0/24      0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       172.27.224.0/20      0.0.0.0/0           
    0     0 AS0_OUT_POST  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_U_SITEB_OUT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  142  8224 ACCEPT     all  --  *      *       192.168.0.0/24       0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       10.10.20.0/24        0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       192.168.77.0/24      0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       172.27.224.0/20      0.0.0.0/0           
    0     0 AS0_OUT_POST  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain AS0_WEBACCEPT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   28  1680 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-after-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-after-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-after-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-after-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-after-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-after-output (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-before-output (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-reject-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-reject-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-reject-output (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-track-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-track-input (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain ufw-track-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

savestheday

Rebuilt the OpenVPN server, still the same issue. Site B can access all of Site A, Site A can ping, telnet, traceroute but no HTTP.

Here is the new IPTables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_ACCEPT  all  --  anywhere             anywhere            
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:openvpn
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:https
AS0_WEBACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_WEBACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:943

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_OUT_S2C  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
AS0_OUT_LOCAL  all  --  anywhere             anywhere            

Chain AS0_ACCEPT (5 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_IN (4 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             OpenVPN-VM2         
           all  --  0.0.0.0              anywhere            
AS0_U_BOOSH_IN  all  --  192.168.77.0/24      anywhere            
AS0_IN_POST  all  --  anywhere             anywhere            

Chain AS0_IN_NAT (1 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x8000000
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_IN_POST (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.0.0/24      
AS0_OUT    all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain AS0_IN_PRE (2 references)
target     prot opt source               destination         
AS0_IN     all  --  anywhere             link-local/16       
AS0_IN     all  --  anywhere             192.168.0.0/16      
AS0_IN     all  --  anywhere             172.16.0.0/12       
AS0_IN     all  --  anywhere             10.0.0.0/8          
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_IN_ROUTE (0 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000000
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_OUT (2 references)
target     prot opt source               destination         
           all  --  anywhere             0.0.0.0             
AS0_U_BOOSH_OUT  all  --  anywhere             192.168.77.0/24     
AS0_OUT_POST  all  --  anywhere             anywhere            

Chain AS0_OUT_LOCAL (1 references)
target     prot opt source               destination         
DROP       icmp --  anywhere             anywhere             icmp redirect
ACCEPT     all  --  anywhere             anywhere            

Chain AS0_OUT_POST (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain AS0_OUT_S2C (1 references)
target     prot opt source               destination         
AS0_OUT    all  --  anywhere             anywhere            

Chain AS0_U_BOOSH_IN (1 references)
target     prot opt source               destination         
AS0_IN_NAT  all  --  anywhere             192.168.0.0/24      
AS0_IN_POST  all  --  anywhere             anywhere            

Chain AS0_U_BOOSH_OUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.0.0/24       anywhere            
ACCEPT     all  --  192.168.77.0/24      anywhere            
ACCEPT     all  --  172.27.224.0/20      anywhere            
AS0_OUT_POST  all  --  anywhere             anywhere            

Chain AS0_WEBACCEPT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere

Here is the output of TCPDUMP from a Mac to 192.168.77.43 (a web server running on port 80)

sudo tcpdump -i any -n -v dst host 192.168.77.43
tcpdump: data link type PKTAP
tcpdump: listening on any, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
12:57:40.757529 IP (tos 0x0, ttl 64, id 40720, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.0.231.58128 > 192.168.77.43.80: Flags [SEW], cksum 0x3185 (correct), seq 4123362743, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 147508652 ecr 0,sackOK,eol], length 0
12:57:40.783092 IP (tos 0x0, ttl 64, id 33724, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.0.231.58128 > 192.168.77.43.80: Flags [.], cksum 0x9b74 (correct), ack 2434086152, win 8192, length 0
12:57:40.784287 IP (tos 0x0, ttl 64, id 15147, offset 0, flags [DF], proto TCP (6), length 456)
    192.168.0.231.58128 > 192.168.77.43.80: Flags [P.], cksum 0xfafe (correct), seq 0:416, ack 1, win 8192, length 416: HTTP, length: 416
	GET / HTTP/1.1
	Host: 192.168.77.43
	Connection: keep-alive
	Cache-Control: max-age=0
	Upgrade-Insecure-Requests: 1
	User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
	DNT: 1
	Accept-Encoding: gzip, deflate, sdch
	Accept-Language: en-US,en;q=0.8
	
12:57:41.076547 IP (tos 0x0, ttl 64, id 50307, offset 0, flags [DF], proto TCP (6), length 456)
    192.168.0.231.58128 > 192.168.77.43.80: Flags [P.], cksum 0xfafe (correct), seq 0:416, ack 1, win 8192, length 416: HTTP, length: 416
	GET / HTTP/1.1
	Host: 192.168.77.43
	Connection: keep-alive
	Cache-Control: max-age=0
	Upgrade-Insecure-Requests: 1
	User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
	DNT: 1
	Accept-Encoding: gzip, deflate, sdch
	Accept-Language: en-US,en;q=0.8
	
12:58:41.146555 IP (tos 0x0, ttl 64, id 44501, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.0.231.58467 > 192.168.77.43.80: Flags [S], cksum 0x09b6 (correct), seq 492872660, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 147566384 ecr 0,sackOK,eol], length 0
12:58:41.154346 IP (tos 0x0, ttl 64, id 52071, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.0.231.58467 > 192.168.77.43.80: Flags [.], cksum 0x644f (correct), ack 2434409758, win 8192, length 0
12:58:41.154763 IP (tos 0x0, ttl 64, id 22315, offset 0, flags [DF], proto TCP (6), length 456)
    192.168.0.231.58467 > 192.168.77.43.80: Flags [P.], cksum 0xc3d9 (correct), seq 0:416, ack 1, win 8192, length 416: HTTP, length: 416
	GET / HTTP/1.1
	Host: 192.168.77.43
	Connection: keep-alive
	Cache-Control: max-age=0
	Upgrade-Insecure-Requests: 1
	User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
	DNT: 1
	Accept-Encoding: gzip, deflate, sdch
	Accept-Language: en-US,en;q=0.8
	
12:58:41.394142 IP (tos 0x0, ttl 64, id 24631, offset 0, flags [DF], proto TCP (6), length 456)
    192.168.0.231.58467 > 192.168.77.43.80: Flags [P.], cksum 0xc3d9 (correct), seq 0:416, ack 1, win 8192, length 416: HTTP, length: 416
	GET / HTTP/1.1
	Host: 192.168.77.43
	Connection: keep-alive
	Cache-Control: max-age=0
	Upgrade-Insecure-Requests: 1
	User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
	DNT: 1
	Accept-Encoding: gzip, deflate, sdch
	Accept-Language: en-US,en;q=0.8

savestheday

Another update:

Today I reversed the roles. I relocated the OpenVPN AS at Site B and had Site A be the client. I have all of the same issues. I am stumped!

The Johan

I would suggest contacting the support ticket system via http://www.openvpn.net > sign in > support.

Present your findings there, and explain what settings your Access Server has now. Personnel there can help you to further diagnose this issue. Off the top of my head I would say this is caused by using NAT in one of the configuration options instead of routing, or there is a static route missing somewhere or traffic is being filtered away at a gateway/firewall system. That pretty much covers all options but it can be logically broken down into pieces to test and eliminate until you're left with one specific cause, which can then be remedied.

savestheday

Will do, thank you Novaflash!