Intermittently can't ping some hosts on client subnet

[sit_chris]

Howdy folks.

Server config (amblet.conf) :

Code: Select all

port 1194
proto udp
dev tun
ca ca.crt
cert amblet.crt
key amblet.key  # This file should be kept secret
dh dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.10.0 255.255.255.0"  # "dear client, please route this network over the vpn (if not ur local net)"
keepalive 10 30
comp-lzo
max-clients 5
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log         openvpn.log
verb 3
client-config-dir ccd
plugin /usr/lib/openvpn/openvpn-auth-pam.so common-auth
client-cert-not-required
username-as-common-name
route 192.168.10.0 255.255.255.0

Client config (ccd/mtw) :

Code: Select all

iroute 192.168.10.0 255.255.255.0

Goal:
I want one single openVPN client (win7) to connect and expose it's local subnet to the server. The local subnet is an office network with workstations, network printers, wireless access points, et cetera. From the openVPN server in the cloud, I want to be able to ping every host residing on the openVPN client's local subnet. This is to allow a Nagios server running on the ec2 instance to monitor IT assets residing in the office network.

I've pretty much accomplished this goal, with one big problem being that not every host inside the office network is reliably pingable. Something is causing an intermittent problem with this. One day, I can ping everything. The next day, only one or two hosts out of 30 some are pingable. Example of a bad day:

Ping network printer 192.168.10.115:

Code: Select all

 ping -c3 192.168.10.115
PING 192.168.10.115 (192.168.10.115) 56(84) bytes of data.
64 bytes from 192.168.10.115: icmp_req=1 ttl=254 time=45.5 ms
64 bytes from 192.168.10.115: icmp_req=2 ttl=254 time=29.7 ms
64 bytes from 192.168.10.115: icmp_req=3 ttl=254 time=33.4 ms

Ping windows server 192.168.10.85:

Code: Select all

ping -c3 192.168.10.85
PING 192.168.10.85 (192.168.10.85) 56(84) bytes of data.

--- 192.168.10.85 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2016ms

On the main office router, I have a static route set up to route vpn traffic to the openVPN client (192.168.10.116). Here is the routing table for the router:

Code: Select all

Destination IP        Subnet Mask			Default Gateway			Hop Count			Interface
10.8.0.0                        255.255.255.0                     192.168.10.116                    1                               eth0
x.x.x.0                     255.255.255.0                     *                                       0                               eth2
192.168.10.0                  255.255.255.0                     *                                       0                               eth0
default                          0.0.0.0                       69.28.35.1                           40                             eth2

Does anything look really wrong?

[sit_chris]

Yes, openVPN works perfectly then without changing anything, the openVPN server can't ping many of the hosts on the office network. The openVPN client connection stays up, but many of the hosts on the client's network are no longer pingable. It's the same hosts that aren't pingable every time. For example, the network printer I talked about (192.168.10.155) is always pingable, both during the problem time and the working fine time.

I did some more testing and I think I found the problem.

I've got two cisco 24 port switches coming off of the main router, and all the network devices feed off those two switches. I traced the cables of the devices that routinely become un-pingable, and found that they all go into switch #2. Switch #2's web interface is not responding. Something's up with switch #2, I may need to upgrade firmware or replace the device!

I was thinking the problem might have been with the VPN subnet that openVPN creates. I've seen in some of the docs that openVPN needs to create a /30 subnet to be compatible with windows. I think that's for TAP setups though...

Your underlying network is unreliable.

Ahh darn, I let my arrogance get in the way. I was hoping openVPN was at fault, not really opening myself to the fact that the problem could be my infrastructure.

Thanks for your help.

[sit_chris]

Here are the versions:

server:

Code: Select all

OpenVPN 2.2.1 x86_64-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] [MH] [PF_INET6] [IPv6 payload 20110424-2 (2.2RC2)] built on Mar 13 2014

client:

Code: Select all

OpenVPN Client 1.5.6

[sit_chris]

Ok, I updated the firmware on the problem switch. The problem with intermittently not being able to ping some hosts still exists, but I discovered that the workstation running the openVPN client can ping all hosts all the time, whereas pinging from the openVPN server cannot ping across the second switch. I can explain it better with a diagram.



Ok so here is what is going on:

• Workstation with openVPN client can ping Workstation
• Workstation with openVPN client can ping both fileservers and the wireless access point.
• openVPN server via Workstation with openVPN client can ping Workstation with openVPN client and Workstation
• openVPN server via Workstation with openVPN client cannot ping either fileserver or the wireless access point

Seems super weird. The workstation running the openVPN client can ping everything, but once openVPN gets in the mix, it's as if switch #1 doesn't know where switch #2 is, or switch #2 gets the ping but sends the pong to the wrong place...

... aaand I think that's exactly what it was. I double checked my route table on my router. It's set to route the destination IP 10.8.0.0 to the gateway 192.168.10.84, which is supposed to be the IP of Workstation with openVPN client. The workstation address is supposed to be mapped to 192.168.10.84 with static DHCP, but it changed on me. That might be the intermittent problem I'm seeing. Originally the workstation was 192.168.10.116 as seen in previous posts. The whole setup is unreliable if the openVPN client keeps switching it's IP address!