[Solved]Open VPN Performance Question

wilsby

Hello

I have connected 2 servers between 2 separate datacenters via a VPN. Both servers have 1000Mbit ethernet connections to a fat pipe

I am experiencing very low bandwidth over VPN between these servers. Before I post my config, I would like to confirm that this is not 'typical' of OpenVPN performance and that the problem is indeed my configuration. I was hoping for performance in the region of 400Mbit/s. Is this a realistic expectation?

Client to Server NO VPN
==================
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1004 MBytes 841 Mbits/sec

# ping
64 bytes from XXX.XXX.XXX.XXX: icmp_req=1 ttl=53 time=25.2 ms
64 bytes from XXX.XXX.XXX.XXX: icmp_req=2 ttl=53 time=25.2 ms
64 bytes from XXX.XXX.XXX.XXX: icmp_req=3 ttl=53 time=25.3 ms
64 bytes from XXX.XXX.XXX.XXX: icmp_req=4 ttl=53 time=25.2 ms

Client to Server with VPN
===================
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.1 sec 41.4 MBytes 34.4 Mbits/sec

# Ping
64 bytes from 172.21.0.1: icmp_req=1 ttl=64 time=25.1 ms
64 bytes from 172.21.0.1: icmp_req=2 ttl=64 time=25.2 ms
64 bytes from 172.21.0.1: icmp_req=3 ttl=64 time=25.1 ms
64 bytes from 172.21.0.1: icmp_req=4 ttl=64 time=25.1 ms

maikcat

if you had posted your config i would see which cipher do you use....

are you using AES?

Michael.

wilsby

I really appreciate help optimizing this VPN where possible. I am no expert.

I require maximum performance because we are about to migrate our services between datacenters.

VPN server config:

port 1194
proto udp
dev-type tun
dev sh0
tun-mtu 1500
mssfix 1450
reneg-sec 0

ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/vpn2.crt
key /etc/openvpn/keys/vpn2.key
dh /etc/openvpn/keys/dh2048.pem
tls-auth /etc/openvpn/keys/ta.key 0

server 172.22.0.0 255.255.0.0
client-config-dir ccd
ifconfig-pool-persist ipp.txt
route 10.5.0.0 255.255.0.0
push "route 172.16.0.0 255.255.0.0"
push "route 10.33.148.0 255.255.255.0"
push "route 10.33.45.128 255.255.255.128"
push "route 10.32.204.0 255.255.255.0"
push "route 10.34.14.96 255.255.255.224"

client-to-client
keepalive 1 7

cipher DES-EDE3-CBC
comp-lzo

persist-key
persist-tun

status /var/log/openvpn/status.log
log /var/log/openvpn/openvpn.log
mute 20
verb 4

script-security 2

txqueuelen 1000

VPN client config:

client jdi-vpn-client
remote-random
remote XXX.XXX.XXX.XXX 1194
proto udp
dev vpn
dev-type tun
cipher DES-EDE3-CBC
comp-lzo yes
script-security 2
nobind
persist-key
persist-tun
cert /etc/openvpn/keys/jdi-vpn-client.crt
key /etc/openvpn/keys/jdi-vpn-client.key
ca /etc/openvpn/keys/ca.crt
tls-auth /etc/openvpn/keys/ta.key 1
status /var/log/openvpn/status.log
log /var/log/openvpn/openvpn.log
verb 3
mute 20
txqueuelen 1000

Server log:

Thu Jul 23 04:17:21 2015 us=478916 XXX.XXX.XXX.XXX:37197 Re-using SSL/TLS context
Thu Jul 23 04:17:21 2015 us=478947 XXX.XXX.XXX.XXX:37197 LZO compression initialized
Thu Jul 23 04:17:21 2015 us=479033 XXX.XXX.XXX.XXX:37197 Control Channel MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Thu Jul 23 04:17:21 2015 us=479059 XXX.XXX.XXX.XXX:37197 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Jul 23 04:17:21 2015 us=479102 XXX.XXX.XXX.XXX:37197 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher DES-EDE3-CBC,auth SHA1,keysize 192,tls-auth,key-method 2,tls-server'
Thu Jul 23 04:17:21 2015 us=479121 XXX.XXX.XXX.XXX:37197 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher DES-EDE3-CBC,auth SHA1,keysize 192,tls-auth,key-method 2,tls-client'
Thu Jul 23 04:17:21 2015 us=479146 XXX.XXX.XXX.XXX:37197 Local Options hash (VER=V4): 'b5edb94e'
Thu Jul 23 04:17:21 2015 us=479170 XXX.XXX.XXX.XXX:37197 Expected Remote Options hash (VER=V4): '53f7fc82'
Thu Jul 23 04:17:21 2015 us=479205 XXX.XXX.XXX.XXX:37197 TLS: Initial packet from [AF_INET]XXX.XXX.XXX.XXX:37197, sid=63721000 64511d3b
Thu Jul 23 04:17:21 2015 us=913342 XXX.XXX.XXX.XXX:37197 VERIFY OK: depth=1, /C=GB/ST=XXXXXXXXX/L=XXXXXXX/O=XXXXXXXXXXXX/CN=XXXXXXXXXXXX_CA/emailAddress=XXXXXXXXXXXXXXXXXXX
Thu Jul 23 04:17:21 2015 us=913605 XXX.XXX.XXX.XXX:37197 VERIFY OK: depth=0, /C=GB/ST=XXXXXXXXX/L=XXXXXXX/O=XXXXXXXXXXXX/CN=jdi-vpn-client/emailAddress=XXXXXXXXXXXXXXXXXXX
Thu Jul 23 04:17:22 2015 us=53300 XXX.XXX.XXX.XXX:37197 Data Channel Encrypt: Cipher 'DES-EDE3-CBC' initialized with 192 bit key
Thu Jul 23 04:17:22 2015 us=53342 XXX.XXX.XXX.XXX:37197 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 04:17:22 2015 us=53365 XXX.XXX.XXX.XXX:37197 Data Channel Decrypt: Cipher 'DES-EDE3-CBC' initialized with 192 bit key
Thu Jul 23 04:17:22 2015 us=53385 XXX.XXX.XXX.XXX:37197 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 04:17:22 2015 us=91596 XXX.XXX.XXX.XXX:37197 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
Thu Jul 23 04:17:22 2015 us=91646 XXX.XXX.XXX.XXX:37197 [jdi-vpn-client] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:37197
Thu Jul 23 04:17:22 2015 us=91845 MULTI: new connection by client 'jdi-vpn-client' will cause previous active sessions by this client to be dropped. Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect.
Thu Jul 23 04:17:22 2015 us=91965 MULTI_sva: pool returned IPv4=172.22.0.6, IPv6=1a00::b8cc:e2d7:227f:0
Thu Jul 23 04:17:22 2015 us=92034 MULTI: Learn: 172.22.0.6 -> jdi-vpn-client/XXX.XXX.XXX.XXX:37197
Thu Jul 23 04:17:22 2015 us=92055 MULTI: primary virtual IP for jdi-vpn-client/XXX.XXX.XXX.XXX:37197: 172.22.0.6
Thu Jul 23 04:17:24 2015 us=550225 jdi-vpn-client/XXX.XXX.XXX.XXX:37197 PUSH: Received control message: 'PUSH_REQUEST'
Thu Jul 23 04:17:24 2015 us=550263 jdi-vpn-client/XXX.XXX.XXX.XXX:37197 send_push_reply(): safe_cap=960
Thu Jul 23 04:17:24 2015 us=550351 jdi-vpn-client/XXX.XXX.XXX.XXX:37197 SENT CONTROL [jdi-vpn-client]: 'PUSH_REPLY,route 172.16.0.0 255.255.0.0,route 10.33.148.0 255.255.255.0,route 10.33.45.128 255.255.255.128,route 10.32.204.0 255.255.255.0,route 10.34.14.96 255.255.255.224,route 172.22.0.0 255.255.0.0,topology net30,ping 1,ping-restart 7,ifconfig 172.22.0.6 172.22.0.5' (status=1)

Client log:

Thu Jul 23 10:17:21 2015 OpenVPN 2.2.1 x86_64-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] [MH] [PF_INET6] [IPv6 payload 20110424-2 (2.2RC2)] built on Dec 1 2014
Thu Jul 23 10:17:21 2015 WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Thu Jul 23 10:17:21 2015 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Thu Jul 23 10:17:21 2015 Control Channel Authentication: using '/etc/openvpn/keys/ta.key' as a OpenVPN static key file
Thu Jul 23 10:17:21 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 10:17:21 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 10:17:21 2015 LZO compression initialized
Thu Jul 23 10:17:21 2015 Control Channel MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Thu Jul 23 10:17:21 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 23 10:17:21 2015 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Jul 23 10:17:21 2015 Local Options hash (VER=V4): '53f7fc82'
Thu Jul 23 10:17:21 2015 Expected Remote Options hash (VER=V4): 'b5edb94e'
Thu Jul 23 10:17:21 2015 UDPv4 link local: [undef]
Thu Jul 23 10:17:21 2015 UDPv4 link remote: [AF_INET]XXX.XXX.XXX.XXX:1194
Thu Jul 23 10:17:21 2015 TLS: Initial packet from [AF_INET]XXX.XXX.XXX.XXX:1194, sid=8c29fadf 200f6392
Thu Jul 23 10:17:21 2015 VERIFY OK: depth=1, /C=GB/ST=XXXXXXXXX/L=XXXXXXX/O=XXXXXXXXXXXX/CN=XXXXXXXXXXXX_CA/emailAddress=XXXXXXXXXXXXXXXXXXX
Thu Jul 23 10:17:21 2015 VERIFY OK: depth=0, /C=GB/ST=XXXXXXXXX/L=XXXXXXX/O=XXXXXXXXXXXX/CN=vpn2/emailAddress=XXXXXXXXXXXXXXXXXXX
Thu Jul 23 10:17:22 2015 Data Channel Encrypt: Cipher 'DES-EDE3-CBC' initialized with 192 bit key
Thu Jul 23 10:17:22 2015 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 10:17:22 2015 Data Channel Decrypt: Cipher 'DES-EDE3-CBC' initialized with 192 bit key
Thu Jul 23 10:17:22 2015 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 23 10:17:22 2015 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
Thu Jul 23 10:17:22 2015 [vpn2] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:1194
Thu Jul 23 10:17:24 2015 SENT CONTROL [vpn2]: 'PUSH_REQUEST' (status=1)
Thu Jul 23 10:17:24 2015 PUSH: Received control message: 'PUSH_REPLY,route 172.16.0.0 255.255.0.0,route 10.33.148.0 255.255.255.0,route 10.33.45.128 255.255.255.128,route 10.32.204.0 255.255.255.0,route 10.34.14.96 255.255.255.224,route 172.22.0.0 255.255.0.0,topology net30,ping 1,ping-restart 7,ifconfig 172.22.0.6 172.22.0.5'
Thu Jul 23 10:17:24 2015 OPTIONS IMPORT: timers and/or timeouts modified
Thu Jul 23 10:17:24 2015 OPTIONS IMPORT: --ifconfig/up options modified
Thu Jul 23 10:17:24 2015 OPTIONS IMPORT: route options modified
Thu Jul 23 10:17:24 2015 ROUTE default_gateway=10.5.0.1
Thu Jul 23 10:17:24 2015 TUN/TAP device vpn opened
Thu Jul 23 10:17:24 2015 TUN/TAP TX queue length set to 1000
Thu Jul 23 10:17:24 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 23 10:17:24 2015 /sbin/ifconfig vpn 172.22.0.6 pointopoint 172.22.0.5 mtu 1500
Thu Jul 23 10:17:24 2015 /sbin/route add -net 172.16.0.0 netmask 255.255.0.0 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 /sbin/route add -net 10.33.148.0 netmask 255.255.255.0 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 /sbin/route add -net 10.33.45.128 netmask 255.255.255.128 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 /sbin/route add -net 10.32.204.0 netmask 255.255.255.0 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 /sbin/route add -net 10.34.14.96 netmask 255.255.255.224 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 /sbin/route add -net 172.22.0.0 netmask 255.255.0.0 gw 172.22.0.5
Thu Jul 23 10:17:24 2015 Initialization Sequence Completed

Performance without VPN:

iperf -c XXX.XXX.XXX.XXX
------------------------------------------------------------
Client connecting to XXX.XXX.XXX.XXX, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.5.167.76 port 32773 connected with XXX.XXX.XXX.XXX port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 974 MBytes 816 Mbits/sec

Performance over VPN:

iperf -c 172.22.0.1
------------------------------------------------------------
Client connecting to 172.22.0.1, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.22.0.6 port 40141 connected with 172.22.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 65.5 MBytes 54.7 Mbits/sec

CPU usage during performance test over VPN:

These are both powerfull machines, and the openvpn process does not exceed 20% cpu usage on either the client or server.

Other information:

The server and client have a 1Gbit interface. The 816Mbits/sec bandwidth test above confirms the 1Gbit interface bandwidth limit of the client.

Server interface
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:25:90:3c:8b:86 brd ff:ff:ff:ff:ff:ff
37: sh0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000
link/none

Client interface
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 42:01:0a:05:a7:4c brd ff:ff:ff:ff:ff:ff
77: vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000
link/none

I notice the client eth0 interface has n mtu of 1460, but the client vpn interface has a mtu of 1500. Could this be an issue? I don't really understand if I need to tweak the mtu settings in the openvpn config.

wilsby

A small revelation that may help diagnose the issue

1) Disabling ciphers does not affect the performance. I still get bad performance around 50Mbit/s

2) I tried switching proto from udp to tcp and I now get over 200Mbit/s over encrypted VPN! WEIRD!

This suggests there may be some udp related performance tuning that needs to be done. The link is very reliable and there is no need for tcp.

On client:
RX: bytes packets errors dropped overrun mcast
3531543295 9450615 0 0 0 0
TX: bytes packets errors dropped carrier collsns
9183078385 7565775 0 0 0 0

On server:
RX: bytes packets errors dropped overrun mcast
2658099258 6775511 0 0 0 0
TX: bytes packets errors dropped carrier collsns
2330135532 7200861 0 0 0 0

maikcat

1) Disabling ciphers does not affect the performance. I still get bad performance around 50Mbit/s

you can check your servers cipher based speed by issuing something like:

openssl speed -evp bf-cbc

and see which alg can perform faster..

2) I tried switching proto from udp to tcp and I now get over 200Mbit/s over encrypted VPN! WEIRD!

maybe your ISP is limiting the rate of udp packets?

also did you try to leave the default MTU settings?

Michael.

wilsby

I tried without the tun-mtu and mssfix settings but get no improvement to bandwitdh. Still around 55Mbit/s. Omitting those settings causes 'replay-window backtrack occurred' entries in the server's openvpn.log

This is a datacenter to datacenter vpn over high speed / low latency link

64 bytes from 172.22.0.1: icmp_req=1 ttl=64 time=25.2 ms
64 bytes from 172.22.0.1: icmp_req=2 ttl=64 time=25.4 ms
64 bytes from 172.22.0.1: icmp_req=3 ttl=64 time=25.4 ms
64 bytes from 172.22.0.1: icmp_req=4 ttl=64 time=25.4 ms

If udp was being limited it must be targetting vpn only traffic.

I moved the server to a different port 50001 (from 1194) but still get the same low performance over proto udp.

I'm not sure how to diagnose this, but I'm sure we must be missing something.

Traffic

maikcat wroteyou can check your servers cipher based speed by issuing something like:

Code:
openssl speed -evp bf-cbc

Nice 8-)

[root@home ~]# openssl speed -evp bf-cbc
Doing bf-cbc for 3s on 16 size blocks: 17297474 bf-cbc's in 3.00s
Doing bf-cbc for 3s on 64 size blocks: 5222132 bf-cbc's in 3.00s
Doing bf-cbc for 3s on 256 size blocks: 1366560 bf-cbc's in 3.00s
Doing bf-cbc for 3s on 1024 size blocks: 347508 bf-cbc's in 3.00s
Doing bf-cbc for 3s on 8192 size blocks: 43443 bf-cbc's in 3.00s
OpenSSL 1.0.2d 9 Jul 2015
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -Wl,-O1,--sort-common,--as-needed,-z,relro -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
bf-cbc           92253.19k   111405.48k   116613.12k   118616.06k   118628.35k
[root@home ~]# openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 9821439 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 2914073 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 256 size blocks: 768388 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 194625 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 24295 aes-256-cbc's in 3.00s
OpenSSL 1.0.2d 9 Jul 2015
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -Wl,-O1,--sort-common,--as-needed,-z,relro -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      52381.01k    62374.81k    65569.11k    66432.00k    66341.55k

Some tips for speed testing:
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

wilsby

Thank you so much traffic

That link Solved the problem as I am now getting iperf values around 1.3Gbit/s!!!!!!!

Holly ****! I love you man!

maikcat

Thank you so much traffic

maybe you though that my AES based question was irrelevant...

anyway i mark this one as solved.

Michael.

wilsby

No maikcat you misunderstand

I appreciate your help but I just want to clarify in case anyone else experiences these performance bottlenecks over high speed VPN.

The cipher type made no difference. I mentioned that disabling cipher with 'cipher none' had no impact on performance.

What did solve the performance problem was the following optimisations for high speed VPN links:

tun-mtu 48000
fragment 0
mssfix 0
sndbuf 12582912
rcvbuf 12582912
replay-window 128
txqueuelen 1000

With these I went from 50Mbit/s to 1.3Gbit/s!!!!!

Traffic

maikcat wrotemaybe you though that my AES based question was irrelevant

Michael is the expert here .. I just try to help 😃

@ wilsby, at least you read the link and found some useful information.

Thanks for letting us know your solution.