I have had help from some knowledgeable people on the OpenVPN mailing list that gave me additional diagnostic tests to consider to pinpoint the root cause of my issue. I will give a summary of those additional suggestions, and what I did to solve this issue for the benefit of anyone else finding this topic in their search for a similar issue/solution.
Additional diagnostic tests to consider; listed by author.
Selva
The configs and routes on server & router look fine, but not clear whether the windows box (client) has correctly set up the pushed route to the 192.168.112.0/24 subnet. Please check the routing table on the windows client.
Gert
Run wireshark/tcpdump on the tun interfaces, see whether packets are going *in* the tunnel (= routing on the client is right) and whether reply packets are coming back in (= routing on the other end is right).
Optionally run wireshark on the remote machine that isn't reachable and see whether packets coming from the server tun actually arrive there, or are eaten by a firewall rule in between, etc.
Morten
What happens when you try a traceroute from the other server .53 to the vpn-servers tunnel ip 10.9.8.1 ?
debbie10t
have you tried (Method 2) adding a route to **192.168.112.53** for 10.9.8.0 to see what happens .. and use tcpdump on the vpn server .. ?
Tests Completed.
Test #1. Verify that the remote Windows client has correctly set up the pushed route to the 192.168.112.0/24 subnet.
Before VPN Connection (Windows client route print)
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.123.2 192.168.123.111 10
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
192.168.56.0 255.255.255.0 On-link 192.168.56.1 276
192.168.56.1 255.255.255.255 On-link 192.168.56.1 276
192.168.56.255 255.255.255.255 On-link 192.168.56.1 276
192.168.123.0 255.255.255.0 On-link 192.168.123.111 266
192.168.123.111 255.255.255.255 On-link 192.168.123.111 266
192.168.123.255 255.255.255.255 On-link 192.168.123.111 266
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 192.168.56.1 276
224.0.0.0 240.0.0.0 On-link 192.168.123.111 266
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 192.168.56.1 276
255.255.255.255 255.255.255.255 On-link 192.168.123.111 266
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.123.2 Default
===========================================================================
After VPN Connection (Windows client route print)
C:\>ipconfig
Ethernet adapter Local Area Connection 2:
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::44cc:5041:5d8d:ae76%9
IPv4 Address. . . . . . . . . . . : 10.9.8.6
Subnet Mask . . . . . . . . . . . : 255.255.255.252
Default Gateway . . . . . . . . . :
[b]> added routes[/b]
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.123.2 192.168.123.111 10
> 10.9.8.1 255.255.255.255 10.9.8.5 10.9.8.6 31
> 10.9.8.4 255.255.255.252 On-link 10.9.8.6 286
> 10.9.8.6 255.255.255.255 On-link 10.9.8.6 286
> 10.9.8.7 255.255.255.255 On-link 10.9.8.6 286
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
192.168.56.0 255.255.255.0 On-link 192.168.56.1 276
192.168.56.1 255.255.255.255 On-link 192.168.56.1 276
192.168.56.255 255.255.255.255 On-link 192.168.56.1 276
> 192.168.112.0 255.255.255.0 10.9.8.5 10.9.8.6 31
192.168.123.0 255.255.255.0 On-link 192.168.123.111 266
192.168.123.111 255.255.255.255 On-link 192.168.123.111 266
192.168.123.255 255.255.255.255 On-link 192.168.123.111 266
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 192.168.56.1 276
224.0.0.0 240.0.0.0 On-link 192.168.123.111 266
> 224.0.0.0 240.0.0.0 On-link 10.9.8.6 286
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 192.168.56.1 276
255.255.255.255 255.255.255.255 On-link 192.168.123.111 266
> 255.255.255.255 255.255.255.255 On-link 10.9.8.6 286
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.123.2 Default
===========================================================================
This appears to be correct. The Windows client has received the pushed route.
Test #2. Run Wireshark on the TUN interfaces and see whether packets are going into the tunnel.
Wireshark on TUN interface of remote Windows Client during ping to 192.168.112.53
No. Time Source Destination Protocol Info
1 16:14:13.608219 10.9.8.6 192.168.112.53 ICMP Echo (ping) request (id=0x0001, seq(be/le)=21/5376, ttl=128)
Frame 1: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: 00:ff:dc:92:57:17 (00:ff:dc:92:57:17), Dst: 10.9.8.5 (00:ff:dd:92:57:17)
Internet Protocol, Src: 10.9.8.6 (10.9.8.6), Dst: 192.168.112.53 (192.168.112.53)
Internet Control Message Protocol
No. Time Source Destination Protocol Info
2 16:14:13.665459 10.9.8.1 10.9.8.6 ICMP Destination unreachable (Host administratively prohibited)
Frame 2: 102 bytes on wire (816 bits), 102 bytes captured (816 bits)
Ethernet II, Src: 10.9.8.5 (00:ff:dd:92:57:17), Dst: 00:ff:dc:92:57:17 (00:ff:dc:92:57:17)
Internet Protocol, Src: 10.9.8.1 (10.9.8.1), Dst: 10.9.8.6 (10.9.8.6)
Internet Control Message Protocol
< repeated 4 times >
Reading this output, the "host administratively prohibited" phrase caught my attention, as it implied to me that the system knew the proper routing, but that the packet was being administratively blocked (ie., by a firewall). So then I had to figure out which firewall, on which box within the route the packet was taking that was causing the problem. Looking back at my original post I had listed the firewall rules on the VPN box and speculated that I might need an accept rule on the Forward Chain. So I took a shot and changed the default setting on the Forward Chain of my VPN box to ACCEPT ALL, rather than DENY ALL. Once I made that change I could ping any box on the LAN behind the VPN box.
Now, I didn’t want to leave my firewall with a default Accept All setting on the forwarding chain, so I needed to identify a rule specific to the packet type / traffic that I want to allow. Some additional suggestions from folks on the mailing list pointed me in the right direction. The basic solution is to allow traffic using the TUN interface to be forwarded, which means accepting them on the input side of the interface and the output side of the interface (the definition of forwarding). The basic rules are listed below.
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A FORWARD -o tun+ -j ACCEPT
These would need to be placed before any REJECT ALL rule on the forward chain. After adding the rules, the Forward chain on my VPN box now shows:
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1068 115K ACCEPT all -- tun0 any anywhere anywhere
1068 117K ACCEPT all -- any tun0 anywhere anywhere
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-port-unreachable
After this firewall change, I am now able to access all boxes behind my VPN server; other servers and desktop systems. Examples below.
Pings from remote Windows Client connected to VPN.
C:\HomeFiles>ping 192.168.112.53 (other server behind VPN server)
Pinging 192.168.112.53 with 32 bytes of data:
Reply from 192.168.112.53: bytes=32 time=57ms TTL=62
Reply from 192.168.112.53: bytes=32 time=53ms TTL=63
Reply from 192.168.112.53: bytes=32 time=54ms TTL=63
Reply from 192.168.112.53: bytes=32 time=56ms TTL=63
Ping statistics for 192.168.112.53:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 53ms, Maximum = 57ms, Average = 55ms
C:\HomeFiles>ping 192.168.112.101 (desktop box behind VPN server)
Pinging 192.168.112.101 with 32 bytes of data:
Reply from 192.168.112.101: bytes=32 time=57ms TTL=126
Reply from 192.168.112.101: bytes=32 time=59ms TTL=127
Reply from 192.168.112.101: bytes=32 time=55ms TTL=127
Reply from 192.168.112.101: bytes=32 time=57ms TTL=127
Ping statistics for 192.168.112.101:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 55ms, Maximum = 59ms, Average = 57ms
So in conclusion… After I had a working connection directly to my VPN server I have made two changes in my network that now allow full access to all systems behind the VPN server. Note that in my case, my network router is on a different box than my VPN server.
Change #1. Added a static route on the default gateway box that directs the VPN tunnel network traffic back to the VPN box; shown as the second one on the list below.
[2.1.2-RELEASE][admin@pfgateway.mei.lan]/root(1): netstat -rn
Destination Gateway Flags Refs Use Netif Expire
default xxx.yy.190.17 UGS 1 663197177 bge0
10.9.8.0/24 192.168.112.50 UGS 0 0 rl0
xxx.yy.190.16/28 link#1 U 0 49927480 bge0
xxx.yy.190.18 link#1 UHS 0 0 lo0
xxx.yy.190.19 link#1 UHS 0 0 lo0 =>
xxx.yy.190.19/32 link#1 U 0 0 bge0
xxx.yy.190.20 link#1 UHS 0 0 lo0 =>
xxx.yy.190.20/32 link#1 U 0 0 bge0
xxx.yy.190.21 link#1 UHS 0 0 lo0 =>
xxx.yy.190.21/32 link#1 U 0 0 bge0
xxx.yy.190.22 link#1 UHS 0 0 lo0 =>
xxx.yy.190.22/32 link#1 U 0 0 bge0
127.0.0.1 link#6 UH 0 560 lo0
192.168.112.0/24 link#2 U 0 848109484 rl0
192.168.112.11 link#2 UHS 0 99857580 lo0
Change #2. Added two firewall rules to accept the VPN traffic that is forwarded across the VPN box.
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A FORWARD -o tun+ -j ACCEPT
That is all. I hope this detailed description helps anyone else that stumbles across it looking to solve a similar issue. Note that once I identified the root cause issue and the basic solution, I did not complete further diagnostic tests suggested by Gert, Morten, or debbie10t and don’t know what additional help those tests may have provided me, or provide to others in a similar situation. Thanks to all those that gave me guidance in my search for a solution.
Jeff