Very slow transfer speeds

dancos

Hello OpenVPN community,

I've been struggling with this for the past days and I have not made any progress so far. The configuration of OpenVPN was delightfully straight forward and pretty much everything worked out of the box. The only thing wrong with the installation are the transfer speeds. Without OpenVPN "iperf" reports about 300 mbps download speed. As soon as OpenVPN is involved the speed drops to anywhere between 10 and 70 mbps depending on different mtu/crypto configurations.
I've been able to reproduce the issue in a simple adhoc VPN.
Server (Debian 8):

openvpn --dev tun --proto udp --port 11000 --secret static.key --ifconfig 192.168.222.11 192.168.222.10
iperf -c 192.168.222.10

Client (Ubuntu 16.04):

openvpn --dev tun --proto udp --port 11000 --secret static.key --ifconfig 192.168.222.10 192.168.222.11 --remote server.ip
iperf -s

This gives me about 35 mbps on a link that should be 300 mbps. The situation is much worse when using tcp at 2-11 mbps. I've experimented with countless combinations of tun-mtu/fragment/mssfix/sndbuf/rcvbuf and many more, the maximum I've ever reached was about 60-70 mbps by adding the following parameters:

--txqueuelen 15000 --tcp-queue-limit 128 --cipher aes-256-cbc

CPU usage on the client is <50% and on the server it's <20%. I've tried different ports as well (443 for example) and iperf over an SSH tunnel gives me almost the full 300 mbps.

While researching this issue I've found people who have the same problem with no solution, people who had to change a tun-mtu or buf setting and the issue was gone, people who say that this is normal and people who say that the overhead should be max 10%. None of the setting changes I've found have helped.

If anyone has any idea as to what might be causing the slowdown and how we can get at least 100mbps out of the VPN that would be greatly appreciated!

TinCanTech

This may be of some help:
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

dancos

Thank you for your reply. That Wiki entry has helped me run this one-line VPN setup for testing as well as the "--cipher aes-256-cbc" parameter which gave some improvement but the speed remains 70mbps or lower. Changes to mtu values generally made the speed worse, no matter the value.
Even with encryption completely disabled (--cipher none --auth none) the speed remains below 70 mbps (and below 11 mbps using tcp). Outside of the VPN the connection to the same box is still at 300 mbps.

dancos

Maybe as a general question, can this behavior be reproduced by anyone? I.e. using the commands mentioned, can you get almost full link speed?
Searching for a cause for normal behavior would be rather pointless 😃

TinCanTech

If you can describe your network in more detail maybe a bottle neck can be identified.

Personally, I get exactly what I pay for out of my internet while using openvpn.

Pippin

In addition to ^^^^

CPU?
CPU supports AES-NI?

dancos

Thank you for your replies. It's a Linux workstation on a gigabit network behind an ASUS RT-AC68U router that connects to a remote server (ping 17-20 ms, 1 gbps bandwidth) through cable internet (400/40 mbps).
Both CPUs support AES-NI, Server:
Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
Client:
Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz

Even though my upload is vastly slower than the download I get better upload speed: Download I get about 33 mbps (355 mbps without VPN), upload 38 mbps (41 mbps without VPN). The next test is going to be hooking up the workstation directly to the cable modem, I'll post an update if this makes a difference 🙂
/Edit: Download is about 36 mbps without the router, so minimal difference.

dancos

Another interesting test, using the "shaper" option on the server, when it's set to 5000000 (40 mbps) the speed stays the same (36 mbps, same for any shaper value above that). When set to 2500000 (20 mbps) the speed still stays the same, so I get 36 mbps with a 20 mbps limit (and OpenVPN says "Output Traffic Shaping initialized at 2500000 bytes per second").
Reducing shaper to 1250000 (10 mbps) finally limits the bandwidth and I get 10 mbps.

Version information, Server:
OpenVPN 2.3.4 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Nov 12 2015
library versions: OpenSSL 1.0.1t 3 May 2016, LZO 2.08
Client:
OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Feb 2 2016
library versions: OpenSSL 1.0.2g 1 Mar 2016, LZO 2.08

TinCanTech

Server Max. upload speed ?

dancos

Max upload speed of the server would be 1 gbps, 200 mbps guaranteed.

TinCanTech

Try this on your server and then client:
https://sourceforge.net/speedtest/

dancos

I believe this is located in the US while the server and client are in Europe. Still the speeds I see are over 200 mbps. Also, iperf between the two gives me 355 mbps server->client and 41 mbps client->server with a ping of about 20 ms.

Pippin

Asus router has Trend Micro AI?
If so, disable helps?

And yes, test going directly out the "front door"

dancos

Thank you, none of the special functions are enabled. I did test hooking the client directly to the cable modem, it gave me +2-3 mbps but that could also just be variance. Interestingly download seems to be slower than upload even though in terms of bandwidth it should be 10x faster.

Pippin

Sounds like mtu issue.

One time i helped a user having same issue of speed in one direction being bad.
fragment and mssfix did not help any.
Eventually found out that his laptop had the mtu of his network adaptor not standard (1500), it was set to 1300.
Setting it back to default solved the problem.

dancos

Thank you for the hint, I've checked all the MTU values for server, tap devices and client (even router, but taking that out of the equation did not make any difference before). It's always 1500, changing the MTU value up/down when starting the tunnel only decreases performance.

It's hard to say whether it's only slow in one direction or whether the tunnel is just somehow stuck at 40 mbps in both directions since that's also my upload limit.

dancos

Very odd... I left iperf running now with a BW report every 4 seconds. It looks like I periodically get about 100 mbps for a short time then it's back to 30-40:

[ 3] 88396.0-88400.0 sec 17.0 MBytes 35.7 Mbits/sec
[ 3] 88400.0-88404.0 sec 19.1 MBytes 40.1 Mbits/sec
[ 3] 88404.0-88408.0 sec 49.9 MBytes 105 Mbits/sec
[ 3] 88408.0-88412.0 sec 41.1 MBytes 86.2 Mbits/sec
[ 3] 88412.0-88416.0 sec 20.9 MBytes 43.8 Mbits/sec
... more entries ...
[ 3] 88640.0-88644.0 sec 32.8 MBytes 68.7 Mbits/sec
[ 3] 88644.0-88648.0 sec 17.9 MBytes 37.5 Mbits/sec
[ 3] 88648.0-88652.0 sec 22.8 MBytes 47.7 Mbits/sec
[ 3] 88652.0-88656.0 sec 22.4 MBytes 46.9 Mbits/sec
[ 3] 88656.0-88660.0 sec 26.6 MBytes 55.8 Mbits/sec
[ 3] 88660.0-88664.0 sec 54.5 MBytes 114 Mbits/sec
[ 3] 88664.0-88668.0 sec 17.4 MBytes 36.4 Mbits/sec
[ 3] 88668.0-88672.0 sec 17.8 MBytes 37.2 Mbits/sec
[ 3] 88672.0-88676.0 sec 20.8 MBytes 43.5 Mbits/sec

Does anyone know what could explain such variance?

KaszasT

Hello

We have the exact same situation, with worse speeds (iperf 80, 80, 60,40, 20, 0, 0, 0, 20, 60, 80, 40, 0 ,0, etc.). I walked down the same path, nothing helped. Do you got any solution for this problem?

Thanks,
Tibor

ludwig.gramberg

having the same problem here, speeds just make no sense

the scenario (all values in mbit down/up):

client A <---200/10---> vpn-server(100/100) <---300/30---> client B

when client A downloads from client B the speed is limited at precisely 10mbit (avg 9,8)
but that direction should allow for 30mbit since thats the limit here.
then when client B pushed some files to the vpn-server the limit was 20mbit.
doing any up/download outside the realm of openvpn the actual speeds are achieved.

client A and B can easily compute the aes so this should not be the limiting factor, also why does client A not reach at least the 20mbit client B did get? just makes no sense...

I tried the settings which were recommended on the wiki but it did not help.
any advice? also played around with different mtu etc.

server-cfg:

port 1196
proto tcp
dev tun2
ca ca-*****.crt
cert *****.crt
key ****.key  # This file should be kept secret
dh dh2048-****.pem
server 10.10.0.0 255.255.255.0
ifconfig-pool-persist ipp-****.txt
client-config-dir /etc/openvpn/ccd/****
client-to-client
duplicate-cn
keepalive 10 120
cipher AES-256-CBC
comp-lzo no
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status-*****.log
log-append /var/log/openvpn-*******.log

# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 4
mute 10

client-cfg:

remote ***** 1196 tcp-client
persist-key
tls-client
pull
ca /etc/openvpn/*****/ca.crt
dev tun
persist-tun
cert /etc/openvpn/*****/*****.crt
comp-lzo no
nobind
key /etc/openvpn/*****/*****.key
cipher AES-256-CBC

TinCanTech

ludwig.gramberg wrotespeeds just make no sense

ludwig.gramberg wroteproto tcp

:roll: