Losses on VPN client

victorqedu

I use OpenVPN for a few clients and I never had problems until a month ago.
Both server and client are version 2.4.9 and I use them on Windows(server and client).

First, a client started to have big losses but losses were only on the VPN made with OpenVPN, ping between public IP's was perfect.
So after searching for a solution for about a week I switched to Windows VPN and it is fine now.

Today I connected to another client and the problem appeared to this new client.
I deleted all taps from the client and recreated them and I tried to change the MTU in the OpenVPN config but nothing helps.

Server config is:

port 1194
proto tcp
dev tun
ca ..\\keys\\ca.crt
cert ..\\keys\\server.crt
key ..\\keys\\server.key
dh ..\\keys\\dh1024.pem
server 10.8.20.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.16.0 255.255.255.0"
keepalive 10 120
persist-key
persist-tun
status openvpn-status.log
verb 4

Client config is:

client
dev tun
proto tcp
remote x.x.x.x 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca_stack.crt
cert ion_popescu.crt
key ion_popescu.key
verb 4
remote-cert-tls server
ip-win32 ipapi

victorqedu

This is how ping responds, it's a weird recurrence, the problem occurs at fixed time intervals and it manifest by losses or by a big latency.
I numbered the packages between losses or latency time and they are exactly 11 healthy ping packages between the problems.
This does not look like a coincidence, there's something spooky here. 🙂

C:\WINDOWS\system32>ping 192.168.16.251 -t

Pinging 192.168.16.251 with 32 bytes of data:
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=35ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=59ms TTL=127
Reply from 192.168.16.251: bytes=32 time=22ms TTL=127
Reply from 192.168.16.251: bytes=32 time=647ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2112ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2991ms TTL=127
Reply from 192.168.16.251: bytes=32 time=28ms TTL=127
Reply from 192.168.16.251: bytes=32 time=24ms TTL=127
Reply from 192.168.16.251: bytes=32 time=47ms TTL=127
Reply from 192.168.16.251: bytes=32 time=50ms TTL=127
Reply from 192.168.16.251: bytes=32 time=56ms TTL=127
Reply from 192.168.16.251: bytes=32 time=36ms TTL=127
Reply from 192.168.16.251: bytes=32 time=21ms TTL=127
Reply from 192.168.16.251: bytes=32 time=55ms TTL=127
Reply from 192.168.16.251: bytes=32 time=37ms TTL=127
Reply from 192.168.16.251: bytes=32 time=472ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=1960ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2973ms TTL=127
Reply from 192.168.16.251: bytes=32 time=28ms TTL=127
Reply from 192.168.16.251: bytes=32 time=43ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=51ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=454ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Request timed out.
Reply from 192.168.16.251: bytes=32 time=97ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=47ms TTL=127
Reply from 192.168.16.251: bytes=32 time=33ms TTL=127
Reply from 192.168.16.251: bytes=32 time=33ms TTL=127
Reply from 192.168.16.251: bytes=32 time=38ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=531ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2048ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2955ms TTL=127
Reply from 192.168.16.251: bytes=32 time=72ms TTL=127
Reply from 192.168.16.251: bytes=32 time=25ms TTL=127
Reply from 192.168.16.251: bytes=32 time=67ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127
Reply from 192.168.16.251: bytes=32 time=26ms TTL=127
Reply from 192.168.16.251: bytes=32 time=64ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=23ms TTL=127
Reply from 192.168.16.251: bytes=32 time=435ms TTL=127
Reply from 192.168.16.251: bytes=32 time=22ms TTL=127
Request timed out.
Reply from 192.168.16.251: bytes=32 time=102ms TTL=127
Reply from 192.168.16.251: bytes=32 time=69ms TTL=127
Reply from 192.168.16.251: bytes=32 time=73ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=32ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=573ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Request timed out.
Reply from 192.168.16.251: bytes=32 time=94ms TTL=127
Reply from 192.168.16.251: bytes=32 time=18ms TTL=127
Reply from 192.168.16.251: bytes=32 time=33ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=22ms TTL=127
Reply from 192.168.16.251: bytes=32 time=537ms TTL=127
Reply from 192.168.16.251: bytes=32 time=19ms TTL=127
Reply from 192.168.16.251: bytes=32 time=2020ms TTL=127
Reply from 192.168.16.251: bytes=32 time=3002ms TTL=127
Reply from 192.168.16.251: bytes=32 time=18ms TTL=127
Reply from 192.168.16.251: bytes=32 time=77ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=23ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Reply from 192.168.16.251: bytes=32 time=29ms TTL=127
Reply from 192.168.16.251: bytes=32 time=427ms TTL=127
Reply from 192.168.16.251: bytes=32 time=17ms TTL=127
Request timed out.
Reply from 192.168.16.251: bytes=32 time=95ms TTL=127
Reply from 192.168.16.251: bytes=32 time=16ms TTL=127

Ping statistics for 192.168.16.251:
Packets: Sent = 98, Received = 94, Lost = 4 (4% loss),
Approximate round trip times in milli-seconds:
Minimum = 16ms, Maximum = 3002ms, Average = 283ms
Control-C

victorqedu

I updated to OpenPVN 2.5.1 on both client and server but the problem persists.

TinCanTech

victorqedu wrote This does not look like a coincidence, there's something spooky here

On a properly configured network openvpn does not behave like this .. so your network is broken.

Also, I bet you have not even checked your openvpn log files.

victorqedu

You are right, I was thinking to check the log after I finished the tests but I did not, I checked now the server logs and I think I found the error that indicates the source problem but I don't understand it. The message appears many times and it is :

2021-03-30 20:38:45 us=529529 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=52

Maybe increasing tcp-queue-limit on server-side could help? I did not tried this parameter because now the connection seems fine, I don't know what triggers this problem. I tried to transfer a 500 MB file and it went fine, no losses.

https://openvpn.net/community-resources/reference-manual-for-openvpn-2-4/

victorqedu

This is a more complete server log:

2021-03-30 20:38:33 us=691680 x.x.x.x:15994 [<CERTIFICATE NAME>] Peer Connection Initiated with [AF_INET6]::ffff:x.x.x.x:15994
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI_sva: pool returned IPv4=10.8.20.6, IPv6=(Not enabled)
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Learn: 10.8.20.6 -> <CERTIFICATE NAME>/x.x.x.x:15994
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: primary virtual IP for <CERTIFICATE NAME>/x.x.x.x:15994: 10.8.20.6
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 Data Channel: using negotiated cipher 'AES-256-GCM'
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 Data Channel MTU parms [ L:1551 D:1450 EF:51 EB:406 ET:0 EL:3 ]
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2021-03-30 20:38:33 us=691680 <CERTIFICATE NAME>/x.x.x.x:15994 SENT CONTROL [<CERTIFICATE NAME>]: 'PUSH_REPLY,route 192.168.16.0 255.255.255.0,route 10.8.20.1,topology net30,ping 10,ping-restart 120,ifconfig 10.8.20.6 10.8.20.5,peer-id 0,cipher AES-256-GCM' (status=1)
2021-03-30 20:38:33 us=812149 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: bad source address from client [::], packet dropped
2021-03-30 20:38:45 us=529529 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=52
2021-03-30 20:38:46 us=530121 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=40
2021-03-30 20:38:48 us=205740 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=83
2021-03-30 20:38:49 us=206333 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=83
2021-03-30 20:38:51 us=204530 <CERTIFICATE NAME>/x.x.x.x:15994 MULTI: Outgoing TUN queue full, dropped packet len=21

victorqedu

TinCanTech wrote On a properly configured network openvpn does not behave like this .. so your network is broken.

What should I check, I can't imagine something wrong that triggers this behaviour.
The server is a Windows server 2019 with all updates installed, a single network card, no firewall on for the duration of my tests.
As I mentioned in the original post, ping between public IP's is perfect while ping between VPN local networks has the described problem.

victorqedu

Two days have passed and the problem has not reappeared.
But what is the "Outgoing TUN queue"? Is this the packet queue that must leave the server with the destination of the client?
How could this be full when the only thing I do on the VPN is a ping? Is this a TCP packet queue?
I think this could be a bug, it's hard to investigate as I can't reproduce it on demand.

TinCanTech

Check your complete Server and Client logs at --verb 4

chilinux

The server config does not look consistent with running OpenVPN AS. Are you sure this problem is related to using OpenVPN AS?

Try turning on UDP support in the OpenVPN AS control panel and also enable it on the client. Then see if the latency jumps continue.

Also try running a MTR ("Matt's traceroute") from the client to the server's IP address and check the results when the problem is occuring.