What kind of throughput can I expect out of this setup?

blumman

My goal is to optimize my routed TUN setups throughput between site A and site B for normal traffic such as rsync, FTPS, HTTPS, CIFS, SSH etc. I have been using this as reference: https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux#no1

Some facts about the hardware.

OpenVPN server at site A
- OpenVPN version 2.0.9 (can't upgrade)
- libssl0.9.8
- CPU, Intel(R) Atom(TM) CPU D510 @ 1.66GHz (DualCore, no support for AES-NI)
- RAM, 4096MB DDR2 @ 667Mhz
- WAN, 250 Mbits/s downstream and 100 Mbits/s upstream

OpenVPN client at site B
- OpenVPN version 2.0.9 (can't upgrade)
- libssl0.9.8
- CPU, Intel(R) Atom(TM) CPU D405 @ 1.50GHz (SingleCore, no support for AES-NI)
- RAM, 1024MB DDR2 @ 667MHz
- WAN, 100 Mbits/s downstream and 10 Mbits/s upstream

Both sites are using the same ISP and the client will mostly "pull" data from the server i.e. the transfer rate would mainly be 100 Mbits/s between site A and site B. Measured throughputs from site B to A, without tunneling are (according to iperf):

- 96.4 Mbits/s downstream
- 10.6 Mbits/s upstream

I've done some testing with the MTU's but the result make absolutely no sense to me. Take a look at this:

iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.2 port 5001 connected with 172.16.0.2 port 48457
[  4]  0.0-10.1 sec  37.5 MBytes  31.2 Mbits/sec
[  5] local 192.168.1.2 port 5001 connected with 172.16.0.2 port 53760
[  5]  0.0-10.1 sec    136 MBytes    114 Mbits/sec
[  4] local 192.168.1.2 port 5001 connected with 172.16.0.2 port 53819
[  4]  0.0-10.1 sec    202 MBytes    168 Mbits/sec
[  5] local 192.168.1.2 port 5001 connected with 172.16.0.2 port 53853
[  5]  0.0-10.1 sec    263 MBytes    219 Mbits/sec

First run is with MTU size 1500, second 6000, third 9000 and last 12000. From there the throughput just escalates along with the higher MTU size. The actual throughput when doing a drag and drop copy of an ISO file was graphed to 19.34 Mbits/s by Cacti. Can I expect higher throughput with this hardware or does this look like optimal throughput for the hardware used? Changing MTU size does not have a noticeable effect on the throughput nor does a "decrease" in decryption (Blowfish or AES128). Finally, what am I doing wrong with iperf? If I could get reliable results then I could find the optimal MTU.

Server config.

ca ca.crt
cert server.crt
key server.key
dh dh2048.pem
local 192.168.1.2
port X
proto udp
dev tun
persist-tun
persist-key
server 172.16.0.0 255.255.255.0
ifconfig-pool-linear ipp 0
client-to-client
client-config-dir ccd
route 192.168.2.0 255.255.255.0
keepalive 10 120
cipher aes-256-cbc
auth sha256
tls-server
tls-cipher AES256-SHA
tls-auth ta.key 0
verb 6
log-append /var/log/openvpn.log
status openvpn-status.log
max-clients 1
user nobody
group nogroup
comp-lzo yes
tun-mtu 1500
fragment 0
mssfix 0

Client config.

client
dev tun
proto udp
remote X Y
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
comp-lzo yes
verb 6
log-append /var/log/openvpn.log
cipher aes-256-cbc
auth sha256
tls-client
tls-cipher AES256-SHA
tls-auth ta.key 1
nobind
tun-mtu 1500
fragment 0
mssfix 0

janjust

you should easily be able to use the full linespeed (100 Mbp down, 10 Mbps up) with the hardware you describe and without having to resort to "large MTU" tricks.

The reason you're seeing numbers which are higher than linespeed is because you have compression enabled

comp-lzo yes

I'd recommend to turn this off, and play with the encryption settings a bit (e.g. cipher none) to see if you achieve "near-line speed".

blumman

janjust wroteyou should easily be able to use the full linespeed (100 Mbp down, 10 Mbps up) with the hardware you describe and without having to resort to "large MTU" tricks.

That sounds promising. 🙂

janjust wroteThe reason you're seeing numbers which are higher than linespeed is because you have compression enabled

Didn't think of that and by disabling comp-lzo the throughput went up from 19.34 Mbits/s to 29.10 Mbits/s. Figures seems to be "real" in iperf now. Thanks!

By changing cipher, auth to none and some minor changes to the config (on both sides), see below.

#cipher aes-256-cbc
cipher none
#auth sha256
auth none
#tls-server
#tls-cipher AES256-SHA
#tls-auth ta.key 0
comp-lzo none
#fragment 0
#mssfix 0

I went from 29.1 Mbits/s to 35.6 Mbits/s. An improvement but not quite there yet. I'm quite satisfied but if you think it's possible to achieve better throughput I'm all ears. Do you have any other ideas up your sleeves? :mrgreen:

janjust

this may sound weird, but try *lowering* the 'fragment' parameter: I would not go a low as 1200, but there might be a sweet spot between 1350 - 1450

blumman

janjust wrotethis may sound weird, but try *lowering* the 'fragment' parameter: I would not go a low as 1200, but there might be a sweet spot between 1350 - 1450

I've run some additional tests. I see a small increase in throughput while changing the fragmentation values, the higher value the better throughput. I'm seeing these kind of speeds with "auth 256", "chiper cipher aes-256-cbc" and "fragmentation 0".

iperf -c 192.168.2.2 -i 10 -t 60
------------------------------------------------------------
Client connecting to 192.168.2.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.1 port 48343 connected with 192.168.2.2 port 5001
[  3]  0.0-10.0 sec  38.9 MBytes  32.7 Mbits/sec
[  3] 10.0-20.0 sec  39.0 MBytes  32.7 Mbits/sec
[  3] 20.0-30.0 sec  38.4 MBytes  32.2 Mbits/sec
[  3] 30.0-40.0 sec  33.5 MBytes  28.1 Mbits/sec
[  3] 40.0-50.0 sec  38.9 MBytes  32.6 Mbits/sec
[  3] 50.0-60.0 sec  38.5 MBytes  32.3 Mbits/sec
[  3]  0.0-60.1 sec    227 MBytes  31.7 Mbits/sec

When encryption is disabled I see these speeds with "fragmentation 0".

iperf -c 192.168.2.2 -i 10 -t 60
------------------------------------------------------------
Client connecting to 192.168.2.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.1 port 48383 connected with 192.168.2.2 port 5001
[  3]  0.0-10.0 sec  45.3 MBytes  38.0 Mbits/sec
[  3] 10.0-20.0 sec  46.1 MBytes  38.6 Mbits/sec
[  3] 20.0-30.0 sec  45.1 MBytes  37.9 Mbits/sec
[  3] 30.0-40.0 sec  46.1 MBytes  38.6 Mbits/sec
[  3] 40.0-50.0 sec  45.5 MBytes  38.1 Mbits/sec
[  3] 50.0-60.0 sec  43.9 MBytes  36.8 Mbits/sec
[  3]  0.0-60.0 sec    272 MBytes  38.0 Mbits/sec

Encryption still off and "fragmentation 1444" I get these speeds.

iperf -c 192.168.2.2 -i 10 -t 60
------------------------------------------------------------
Client connecting to 192.168.2.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.1 port 48490 connected with 192.168.2.2 port 5001
[  3]  0.0-10.0 sec  50.6 MBytes  42.4 Mbits/sec
[  3] 10.0-20.0 sec  51.9 MBytes  43.5 Mbits/sec
[  3] 20.0-30.0 sec  47.4 MBytes  39.8 Mbits/sec
[  3] 30.0-40.0 sec  50.4 MBytes  42.3 Mbits/sec
[  3] 40.0-50.0 sec  52.0 MBytes  43.6 Mbits/sec
[  3] 50.0-60.0 sec  51.2 MBytes  43.0 Mbits/sec
[  3]  0.0-60.0 sec    303 MBytes  42.4 Mbits/sec

The "mtu-test" gave me this. Seems fine right?

NOTE: Beginning empirical MTU test -- results should be available in 3 to 4 minutes.
NOTE: Empirical MTU test completed [Tried,Actual] local->remote=[1410,1410] remote->local=[1410,1410]

There seems to be leftover processing power as the OpenVPN servers average is at 63% and the OpenVPN client is around the same.

Enabling the encryption with a fragmentation of 1444 brings me back to 32 Mbits/s. Maybe I should be happy with this? It would of course be nice to see close to full line capacity speeds but for now I'm just running batch jobs over night.