Keepalive packet losts, hard reset timeout BUT ping maintain connection

mmbriant

Short story : We have a weird problem. The openVPN keepalive function does not maintain the link up (we see hard reset due to ping timeout) but if we implement a manual ping (linux ping -> ping-response) no packet are dropped and the tunnel is maintained active.

Long story (very very long story):
Were are setting up an Amazon Web Sevice (AWS) virtual machine to provide communications between embedded devices (M2M) located behind firewall (typically cellular private IP or behind industrial LAN with no incoming traffic accepted). Each embedded device open a tunnel to the AWS machine and obtain a local IP (192.168.30.x) to permit communications between devices. We use UDP mode on standard port 1194.

Because of bandwidth limitations, we increased the keepalive to 5 minutes (300 secs) and timeout to 33 minutes (2000 seconds, or 6 x keepalive)

During the test setup, we have 5 devices connected to the openVPN server (but we have no communication between devices, only keepalives)
1- Unit behind a domestic internet by cable service, behind a cheap NAT router -> Experience no drop of tunnel
2- Unit on a public IP address in industrial environment, no filtering, no NAT -> Experience no drop of tunnel
3- Unit on private IP address in industrial environment (behind NAT and Fortinet filtering) -> Experience drop of tunnel, LOST of keepalive packets
4- Unit on public IP from wireless provider (3G) -> Experience drop of tunnel, LOST of keepalive packets
5- Unit on private IP from wireless provider (same 3G supplier) -> Experience drop of tunnel, LOST of keepalive packets

Running tcpdump on both the embedded clients and the AWS server, we see the following for units 3/4/5 (that experience the openVPN tunnel drop)
Server send keepalive to each unit, unit sees the the keepalive. This is repeated every 5 minutes
Each unit send keepalive to server, server sees the keepalive. This is also repeated every 5 minutes
Once per hour, each client renegotiate the TLS keys with the server (generating more traffic).

Then... at some point for a specific unit:
Server send a keepalive to that unit, unit DOES NOT see the the keepalive
The unit send keepalive to server, server sees the keepalive.
-> This may occur once, then behaviour resume normally (unit see the other keepalive).
-> Maybe 2-3 or 4 keepalive are missed by the client BUT one finally get though...
-> Client misses 6 consecutive keepalive packets from the server... then close the IP port, open a new one and send a "P_CONTROL_HARD_RESET_CLIENT_V2" frame

The above seem RANDOM behaviour on units 3/4/5. So tunnel goes down maybe once per hour, 2 hours, or up to 6 hours...

Note : The "missing" of the server sent keepalive is confirmed by the tcpdump of the server (seing the packet going out) and absence of incoming packet both in the embedded device tcpdump log AND with external mirroring switch and a wireshark capture.

After trying for a week to diagnose the problem, we implemented in units 3, 4 and 5 (all that experience the packet drop) a manual ping to the AWS server (192.168.30.1) with repetition rate of 4 minutes (240 seconds). We expected this to have the effect of blocking BOTH keepalive (server->client and client->server) because ping is bi-directional (client send ping request and receive ping response from server).

Indeed we see that the mechanism works... What is fantastic is that NO PING packet are lost (we tracked over 1 day of exchange). Because of those PING, no keepalive are sent and no timeout occurs (no hard reset of the tunnel).

We just cannot understand WHY some encrypted packets (openVPN keepalive) could be dropped by some devices (fortigate filtering or GSM provider firewall) but the encrypted PING packets are never dropped.

This is troublesome because:
-it may indicate a bigger problem (that will cause us problem eventually);
-it also use almost double the bandwidth (keepalive packets are 80/83 bytes long and ping/ping-response are 151/148 bytes.
-it mean that we will have to "customize" openvpn clients in our devices to implement the manual ping service INSTEAD of the built-in keepalive.

Thanks (and sorry again but I could not make this stuff shorter !)

Michel

TinCanTech

NAT firewalls have a timeout for all packets .. if you do not keep the path open with ping (--keepalive 10 120 is sufficient) then the client side NAT will close and the server will drop the client.

mmbriant

This had been considered but we don't think it's the cause of the problem because :

- NAT closing possibility does not explain why unit #1 (behind NAT) does not suffer for tunnel drop

- For units 3/4/5, as described in symptoms, sometimes packets are not received by client but NAT did not close the port because we see, later on, other keepalive packets from server being received on the same port.

- keepalive of 10 120 "maybe" sufficient but will use MUCH too much traffic. Sending packets each 10 seconds (one each way) over cellular network will end up having a heafty bill in some countries (rough calc = 80 bytes x 2 directions x 86400 secs/day / 10 seconds x 31 days = 42.85 MB. This may sound small to some (with their home limit of 300GB) but over canadian cellular network... Also, I'm really not sure that it will solve the problem (see last point)

Finally, why sending one PING inside the tunnel, and getting one PING response inside the tunnel, each 240 seconds (4 minutes) correct the problem?

TinCanTech

mmbriant wrote This had been considered but we don't think it's the cause of the problem because :

- NAT closing possibility does not explain why unit #1 (behind NAT) does not suffer for tunnel drop

Not all routers are equal ..

mmbriant wrote why sending one PING inside the tunnel, and getting one PING response inside the tunnel, each 240 seconds (4 minutes) correct the problem?

Is that with --keepalive set also ?

The simple fact is, there is a lot of equipment beyond your control and even knowledge of, you will just have to experiment. But huge periods between any kind of keep alive packet will expire NAT'ed ports.

mmbriant

mmbriant wrote: ↑Tue Oct 17, 2017 5:56 pm
why sending one PING inside the tunnel, and getting one PING response inside the tunnel, each 240 seconds (4 minutes) correct the problem?

TomCamTech wrote:
Is that with --keepalive set also ?

Yes --keepalive are set but with 300 2000 parameters. Since we send ping each 240 seconds (and server respond also every 240 seconds), the openVPN built-in keepalive never has to send it's own "ping"

And again, the problem is not the NAT closing the port. If the NAT was closing the port, there would be no problem understanding what is happening. The router/firewall/other device does NOT close the port because sometimes the openVPN ping goes through.

I'm not familiar with rules here but can I send a small 23KB .pcap (tcpdump / wireshark) capture of what is actually is happening.... and would this help ?

TinCanTech

mmbriant wrote TomCamTech wrote

who ?

mmbriant

Oups :o ... sorry. My fingers kind of had their own mind...

Of course I meant TinCanTech and not TomCamTech !