Gigabit Server but cannot exceed more than 160mbps

rainbow6

Hi,
I have a gigabit server running openvpn proxy server serving a 200 road warrior users. Normal test using wget and http, traffic can achieve 700mbps without openvpn.
However with more than 100 clients connecting to it, total bandwidth tops at 160Mbps. This is running in Centos 5.6.

Server specs: 8GB RAM, AMD Quad Cores, SSD Drive.

Sysctl Network values as follows:

net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic
net.core.wmem_default = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.ipv4.tcp_timestamps = 0

Partial Openvpn Server Config (hidden are the ipconfig etc..):

keepalive 10 300
ping-timer-rem
persist-tun
persist-key
max-clients 200

float
;mssfix max
mute-replay-warnings

# Run OpenVPN as a daemon and drop privileges to user/group nobody user nobody
group nobody
daemon

status status.log 30
status-version 2

log  openvpn.log
verb 2

sndbuf 655360
rcvbuf 655360
txqueuelen 1000

With more than 100 users connected, top shows that memory and CPU usage is still reasonable.
Cpu(s): 26.2%us, 29.2%sy, 0.0%ni, 33.6%id, 0.0%wa, 1.7%hi, 9.4%si, 0.0%st
Mem: 7918596k total, 2372548k used, 5546048k free, 58528k buffers
Swap: 3604472k total, 0k used, 3604472k free, 332068k cached

From what I can observed is that once it reaches the 160Mbit, every users will be fighting for the available bandwidth within the max speed.
MTU used is default value.

Really appreciate if anyone can tell me what are the best optimum settings (linux and openvpn) to utilize gigabit (server) transfer to road warriors (internet). even reaching 300Mbps is sufficient enough.

TIA

janjust

this is a known problem and thus far, there is very little that can be done about it. Take a look at
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux
which is a result of my initial investigations: it is possible to optimize a single point-to-point link, but it is NOT possible to achieve more than 160 Mbps using a "regular" client/server setup. Since then some students have performed additional tests for me, but the conclusions are still the same.

The technical reason behind this is that encryption/decryption routines are not very efficient or fast when encrypting/decrypting packets of only 1500 bytes. Unless someone can come up with a more efficient way to do this then I'm afraid that this is about the maximum speed you'll get out of a VPN setup. Perhaps an SSL accelerator card might help here, but I've not been able to test that. As an SSL accelerator card costs only ~ $50 I'd say that's worth the effort.

Do note that commercial VPNs suffer the same problem: if you want gigabit IPsec speed you pay TOP dollar.

maikcat

maybe there is a limitation on tun/tap device itself...

Michael.

janjust

nope - turn off encryption and signing and the speed goes up to 400+ Mbps.

rainbow6

In another scenario, i started another openvpn on another port which created another tun devices, tun1. While tun0, with 150 users were maxing out 160mbit and users connecting to tun1 still able to max out the bandwidth, easily topping 40mbit for the 10 additional users.

Is tun driver flawed when it comes to gigabit? I remember during the old days, the tun/tap drivers showed a 10mbit connection when it initializes itself in windows environment.

I will be loading users over 3 tun devices and see if there is a significant improvement over the bandwidth utilization. If it is breaking the total bandwidth over 160mbit with encryption enabled, then the main culprit capping the bandwidth is the tun driver.

maikcat

@JJK

that means this is outdated?

http://forums.openvpn.net/topic7826.html

Michael.

rainbow6

Not sure whether its outdated or not BUT pretty sure that tun device is not displaying correct value. ethtool still giving 10mbit when its pushing 160 mbit, unless its trying to say 10mbit per client.

rainbow6

rainbow6 wroteNot sure whether its outdated or not BUT pretty sure that tun device is not displaying correct value. ethtool still giving 10mbit when its pushing 160 mbit, unless its trying to say 10mbit per client.

Preliminary result of the tests and hope that others will be able to use this as well.
This is what I have done to bypass the limitations on bandwidth saturation using Gigabit, the tests were conducted on Centos.
1. Allocated 4 additional IP aliases to the main ETH0
2. Create openvpn1.conf to openvpn5.conf, each with a different IP addresses but listening on the same port.
3. Starts OpenVPN and now having 5 addresses in 5 diff tun devices, listening to user connections
4. Each tun/config files is configured with a max of 60 users (/30).
5. All tun devices is routed back to the internet using ETH0.

With 220 users connected and spread over 5 tun devices, it resulted in the following:

bwm-ng v0.5 (probing every 0.500s), press 'h' for help
input: libstatgrab type: rate
  -       iface                  Rx                   Tx                Total
  ===========================================================================
            lo:           3.24 KB/s            3.24 KB/s            6.48 KB/s
          eth0:       26168.58 KB/s        28198.29 KB/s        54366.87 KB/s
          tun0:         635.23 KB/s         4780.21 KB/s         5415.44 KB/s
          tun1:         480.14 KB/s         4607.91 KB/s         5088.05 KB/s
          tun2:         569.26 KB/s         2545.08 KB/s         3114.35 KB/s
          tun3:         831.13 KB/s         3634.40 KB/s         4465.52 KB/s
          tun4:         840.13 KB/s         6189.81 KB/s         7029.95 KB/s
          tun5:           0.00 KB/s            0.00 KB/s            0.00 KB/s
          tun6:         172.40 KB/s         1149.33 KB/s         1321.73 KB/s
  ---------------------------------------------------------------------------
         total:       29700.11 KB/s        51108.27 KB/s        80808.38 KB/s

With this output, it seems that the bottleneck is tun device itself. Each tun can handle only 160mbit transfer rate. Though the tests is not conducted in lab environment, the real world production showed that this can be a workaround to those facing the same issues. It probably useful for the road warrior setup than the LAN setup. It probably be another 10 years for mobile users to achieve gigabit connectivity. At least my users are now happy with the performance.

I hope that OpenVPN team can run the tests more thoroughly and update us on this.

http://status.bolehvpn.net/status/crysis.html

janjust

@JJK

that means this is outdated?

http://forums.openvpn.net/topic7826.html

no I don't think so, as it depends a lot on the testing setup. Using an AESNI capable CPU will certainly help. Splitting the traffic over multiple tun adapters will also certainly help. The biggest hurdle seems to be in forwarding traffic from clients behind the openvpn endpoints: I've found ways to increase throughput between the endpoints, but as soon as I start adding clients behind the openvpn endpoints themselves (e.g. when using lan-to-lan VPNs) then performance drops again to ~160 Mbps.

Not sure whether its outdated or not BUT pretty sure that tun device is not displaying correct value. ethtool still giving 10mbit when its pushing 160 mbit, unless its trying to say 10mbit per client.

ethtool cannot determine the link speed for tun or tap device and hence returns the system "default" , which is 10 mpbs; the tun device is capable of achieving much higher speeds: I've pushed 3+ Gbps over a single tun device (without encryption).

yourfreevpn

rainbow6 wrote
rainbow6 wroteNot sure whether its outdated or not BUT pretty sure that tun device is not displaying correct value. ethtool still giving 10mbit when its pushing 160 mbit, unless its trying to say 10mbit per client.
Preliminary result of the tests and hope that others will be able to use this as well.
This is what I have done to bypass the limitations on bandwidth saturation using Gigabit, the tests were conducted on Centos.
1. Allocated 4 additional IP aliases to the main ETH0
2. Create openvpn1.conf to openvpn5.conf, each with a different IP addresses but listening on the same port.
3. Starts OpenVPN and now having 5 addresses in 5 diff tun devices, listening to user connections
4. Each tun/config files is configured with a max of 60 users (/30).
5. All tun devices is routed back to the internet using ETH0.

With 220 users connected and spread over 5 tun devices, it resulted in the following:
bwm-ng v0.5 (probing every 0.500s), press 'h' for help
input: libstatgrab type: rate
  -       iface                  Rx                   Tx                Total
  ===========================================================================
            lo:           3.24 KB/s            3.24 KB/s            6.48 KB/s
          eth0:       26168.58 KB/s        28198.29 KB/s        54366.87 KB/s
          tun0:         635.23 KB/s         4780.21 KB/s         5415.44 KB/s
          tun1:         480.14 KB/s         4607.91 KB/s         5088.05 KB/s
          tun2:         569.26 KB/s         2545.08 KB/s         3114.35 KB/s
          tun3:         831.13 KB/s         3634.40 KB/s         4465.52 KB/s
          tun4:         840.13 KB/s         6189.81 KB/s         7029.95 KB/s
          tun5:           0.00 KB/s            0.00 KB/s            0.00 KB/s
          tun6:         172.40 KB/s         1149.33 KB/s         1321.73 KB/s
  ---------------------------------------------------------------------------
         total:       29700.11 KB/s        51108.27 KB/s        80808.38 KB/s
With this output, it seems that the bottleneck is tun device itself. Each tun can handle only 160mbit transfer rate. Though the tests is not conducted in lab environment, the real world production showed that this can be a workaround to those facing the same issues. It probably useful for the road warrior setup than the LAN setup. It probably be another 10 years for mobile users to achieve gigabit connectivity. At least my users are now happy with the performance.

I hope that OpenVPN team can run the tests more thoroughly and update us on this.

http://status.bolehvpn.net/status/crysis.html

I think the bottleneck is not the tun device but the CPU. A single openvpn server can only use a single core of your processor thereby limiting how much data it can process. AFAIK openvpn is still not multithreaded. When you created multiple vpn servers listening on different IPs you are distributing the work on all the cores of your CPU. I already read some posts just like janjust that they have achieve gigabit speeds on a single tun device with encryption turned off.