High latency and slow transfer speeds over openvpn tunnel

cousine

Hi,

I have a server running openvpn server within a docker container, testing the transfer performance to the server using iperf gives full bandwidth (1Mbps down, 0.5Mbps up) and (80 ~ 85ms) without VPN, however once I connect using VPN my latency jumps around 40ms (120 ~ 125ms) and my transfer slows down to (0.2 Mbps down, 0.5 Mbps up).

I've tried to change the configuration to use tcp instead of udp, change the port numbers and use different connections (and ISPs) to rule out packet shaping and throttling, and still hit the same brick, perfect transfers off vpn, very bad transfer while connected to vpn.

Another very weird symptom is that sometimes (very rarely and after a lot of trials) I could connect one client to the server (from the same local network) and that client would have the full bandwidth however any other client (from the same local network) would still suffer the slow speeds.

Here is my Server configuration: (nothing special at all)

server 192.168.255.0 255.255.255.0
verb 3
key /etc/openvpn/pki/private/[HOST].key
ca /etc/openvpn/pki/ca.crt
cert /etc/openvpn/pki/issued/[HOST].crt
dh /etc/openvpn/pki/dh.pem
tls-auth /etc/openvpn/pki/ta.key
key-direction 0
keepalive 10 60
persist-key
persist-tun

proto udp
# Rely on Docker to do port mapping, internally always 1194
port 1194
dev tun0
status /tmp/openvpn-status.log

user nobody
group nogroup
#push dhcp-option DNS 8.8.8.8
#push dhcp-option DNS 8.8.4.4
push dhcp-option DNS 208.67.222.222
push dhcp-option DNS 208.67.220.220
route 192.168.254.0 255.255.255.0

sndbuf 0
rcvbuf 0
push "sndbuf 0"
push "rcvbuf 0"

And here is a sample client configuration:

client
nobind
dev tun
remote-cert-tls server

remote HOST 1194 udp


<key>
KEY HERE
</key>
<cert>
CERTIFICATE HERE
</cert>
<ca>
CERTIFICATE HERE
</ca>
<tls-auth>
KEY HERE
</tls-auth>
key-direction 1

redirect-gateway def1

I also tried adjust the MTU and MSS on both server and client to no avail.

Note: this configuration used to work fine until around 2 weeks ago after I had to move my server to a different datacenter.

Thanks

Traffic

cousine wroteNote: this configuration used to work fine until around 2 weeks ago after I had to move my server to a different datacenter

:idea: .. I wonder what else changed .. :ugeek:

cousine

Supposably nothing, I used the same snapshot so only the datacenter, and I thoroughly tested the transfer from/to that datacenter and that specific server to make sure its not ISP related

cousine

Quick update, somehow I managed to get one client certificate working, so the next thing I tried was moving the certificate to a different machine on the same local network which worked fine as well, however when I tried another certificate the transfer rates were unchanged.

So right now I have 1 client certificate working fine, but all the others are showing the same issues above (this happened before and eventually the working certificate fails and reverts back)

Traffic

Traffic wrote
cousine wroteNote: this configuration used to work fine until around 2 weeks ago after I had to move my server to a different datacenter
:idea: .. I wonder what else changed .. :ugeek:

cousine wroteSupposably nothing ...

Check that thoroughly .. openvpn does not throttle.

terrorfrog

ok here my 4 cents

i cant say this often enough - check for povider filtering.
you said you moved to a new datacenter - which means you get different routes.

which means somewhere in between maybe filtering udpo pakcets and slow em down.
why? well many reason but #1 is prevent voiip communication, which seems to become a sport at isps
mayn of those are telecom too and they really dont like competition on "their" lines.

now to trace the problem you need winmtr or on linux mtr (winmtr please use the version from the website not sourceforge, later one is very old) make a trace from both sides same time
make 2 simulatious traces on each side, one on the vpn ips, one on the public ips (as far you can go, usually your firewall at client side and openvpn ip/firewall on the other)
dont wonder if you see different routes thats normal for big datacenters

now first indication is different latency by much.
if openvpn is jumpy but regular ping is not it means the fault is somewhere on the vpn side.
that is either indeed some problems on your server (not very likely) or filtering is active

is problems are on both pings (outside vpn and inside) then theres is a packetloss problem somewhere between datacenter and you.
the only thing you can do now is to identify the HOP where the loss happens , find their phonenumber and call em - yes that works i do at least 10 calls like this a year, you would wonder how long big isp need to realize their 10gbit line isnt working properly 🙂

now in case of filtering - well now youre in trouble but dont give up just yet.
first you can try switch ports and see if that helps, some do filtering on higher ports only. try togo to http, smtp, dns or something else very common.
second try to switch to tcp - that usually shoudl solve it.
another way is to increase mtu and do some other things in the tcpip stack BUT i recommend to avoid this if possible
however there plenty of guide what you can do about that...