Expert needed: OpenVPN bonding throughput less than expected

This forum is for admins who are looking to build or expand their OpenVPN setup.

Moderators: TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech

Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Post Reply
devnull
OpenVpn Newbie
Posts: 5
Joined: Thu Oct 24, 2013 5:29 pm

Expert needed: OpenVPN bonding throughput less than expected

Post by devnull » Thu Oct 24, 2013 6:52 pm

Hello!

(Please forgive my bad English)

I have read OpenVPN manual three times, tested every option that could help but I am stuck... (Google did not help)

At home, I have the luck to have two (very) good ADSL lines (120 ft from the "remote concentrator").

Each ADSL line (PPPOE) is stable and fast : 20 mbit/s IN 1 mbit/s OUT.

A small Linux "box" (Atom 1.8 Ghz, 2 cores, HT) is acting as a router/firewall (Debian 7, lastest OpenVZ kernel) and traffic shaping is used (htb) to keep personnal VOIP clean.

So, I have "bonded" the two ADSL lines with OpenVPN to a (fat) remote server (at work): 1 gbit/s NIC, throughput can stand 600 mbit/s IN or OUT without problem.

OpenVPN Bonding is working well (mode 0, balance-rr) but download speed is "hitting a wall" : ~23 Mbit/s (while I could expect more than 35 Mbit/s).

If I did not make any error, a MTU of 1448 bytes is perfect to exactly fit 30 ATM cells: 29 cells * 48 bytes + last cell (48 bytes + 8 bytes SAR trailer) = 1448 bytes.

Tcpdump (-n -v -e -i vmbrX port 31415 and greater 1462 (1462 - 14 bytes for Ethernet)) output confirms this :

Code: Select all

ethertype IPv4 (0x0800), length 1462: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1448)
    ip.addr.dest 31415 > ip.addr.source.34483: UDP, length 1420
At first, I thought than the Atom processor was to slow to handle LZO compression, encryption and HMAC verification so I disabled everything: no TLS (static key), LZO (comp-lzo no), replay (no-replay), auth (none), cipher (none).

That did not help.

Now, (while downloading) I suspect ACK packets (inside the VPNs) to completly fill up the upload of each ADSL line but I also need to double-check with wireshark if the problem could be related to TCP DUP ACK/TCP Retransmission.

Below are the configuration files in use.

Any help on this problem would be greatly appreciated. :)

Thanks!

OpenVPN clients:

Code: Select all

remote xxx.xxx.xxx.xxx 31415
proto udp
nobind
dev bond-client1
dev-type tun
fragment 1420
mssfix
sndbuf 262144
rcvbuf 262144
txqueuelen 256
ping-restart 0
persist-tun
persist-key
script-security 2
cd /home/master/root/etc/openvpn/bond-client-X/
log log/openvpn.log
nice -20
fast-io
verb 4
status log/status.log 300
comp-lzo no
secret keys/static.key
auth none
cipher none
no-replay
up bin/up.sh
down bin/down.sh
OpenVPN servers:

Code: Select all

local xxx.xxx.xxx.xxx
proto udp
port 31415
dev bond-server-X
dev-type tun
fragment 1420
mssfix
sndbuf 262144
rcvbuf 262144
txqueuelen 256
ping-restart 0
persist-tun
persist-key
script-security 2
cd /home/master/root/etc/openvpn/bond-server-X/
log log/openvpn.log
nice -20
fast-io
verb 4
status log/status.log 300
comp-lzo no
secret keys/static.key
auth none
cipher none
no-replay
up bin/up.sh
down bin/down.sh
bond0 interface (client & server)

Code: Select all

iface bond0 inet static
        pre-up /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client-1
        pre-up /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client-2
        post-up /etc/init.d/openvpn start bond-client1 bond-client-2
        pre-down /etc/init.d/openvpn stop bond-client1 bond-client-2
        pre-down /bin/sleep 5
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client-1
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client-2
        post-down /sbin/rmmod bonding
        bond_mode balance-rr
        slaves bond-client-1 bond-client-2
        address 10.10.10.1
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255

User avatar
maikcat
Forum Team
Posts: 4200
Joined: Wed Jan 12, 2011 9:23 am
Location: Athens,Greece
Contact:

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by maikcat » Fri Oct 25, 2013 6:25 am

are you sure your bonding setup works?

AFAIK you could only achieve bonding by using tap interface devices
which they operate at layer2, at least i have tested this and was working...
aa:~ # modinfo bonding
filename: /lib/modules/3.4.47-2.38-desktop/kernel/drivers/net/bonding/bonding.ko
alias: rtnl-link-bond
author: Thomas Davis, tadavis@lbl.gov and many others
description: Ethernet Channel Bonding Driver, v3.7.1
curious though, can you post also the contents of your up/down scripts?

Michael.
Amiga 500 , Zx +2 owner
Long live Dino Dini (Kick off 2 Creator)

Inflammable means flammable? (Dr Nick Riviera,Simsons Season13)

"objects in mirror are losing"

devnull
OpenVpn Newbie
Posts: 5
Joined: Thu Oct 24, 2013 5:29 pm

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by devnull » Sat Oct 26, 2013 5:49 pm

Many thanks for your answer! :)

I think I have found the problem...

=> I need to test, double-check, etc.

And when all is running smoothly, I will post every config files in this topic.

By the way, bonding over tun interfaces works!

devnull
OpenVpn Newbie
Posts: 5
Joined: Thu Oct 24, 2013 5:29 pm

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by devnull » Thu Jan 23, 2014 8:38 pm

Hello! :D

So, problem (almost) solved, here we go:

Server config:

Code: Select all

local <server_ip_address_1>
proto udp
port 1194
dev bond-server1
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
auth none
cipher BF-CBC
replay-window 256 60
tls-server
ca ca.crt
dh dh4096.pem
cert server-1.crt
key server-1.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 0

Code: Select all

local <server_ip_address_2>
proto udp
port 1194
dev bond-server2
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
auth none
cipher BF-CBC
replay-window 256 60
tls-server
ca ca.crt
dh dh4096.pem
cert server-2.crt
key server-2.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 0

Code: Select all

# auto bond0
iface bond0 inet static
        pre-up    /sbin/modprobe bonding
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-server1
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-server2
        post-up   /etc/init.d/openvpn start bond-server1 bond-server2
        pre-down  /etc/init.d/openvpn stop  bond-server1 bond-server2
        post-down /bin/sleep 5
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-server1
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-server2
        post-down /sbin/rmmod bonding
        bond_mode balance-xor
        bond_xmit_hash_policy layer3+4
        slaves bond-server1 bond-server2
        address 10.10.10.2
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255
Client config:

Code: Select all

remote <server_ip_address_1> 1194
proto udp
nobind
dev bond-client1
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
client
auth none
cipher BF-CBC
replay-window 256 60
tls-client
ca ca.crt
cert client-1.crt
key client-1.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 1
ns-cert-type server

Code: Select all

remote <server_ip_address_2> 1194
proto udp
nobind
dev bond-client2
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
client
auth none
cipher BF-CBC
replay-window 256 60
tls-client
ca ca.crt
cert client-2.crt
key client-2.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 1
ns-cert-type server

Code: Select all

# auto bond0
iface bond0 inet static
        pre-up    /sbin/modprobe bonding
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client1
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client2
        post-up   /etc/init.d/openvpn start bond-client1 bond-client2
        pre-down  /etc/init.d/openvpn stop  bond-client1 bond-client2
        post-down /bin/sleep 5
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client1
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client2
        post-down /sbin/rmmod bonding
        bond_mode balance-xor
        bond_xmit_hash_policy layer3+4
        slaves bond-client1 bond-client2
        address 10.10.10.1
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255
Notes:

- Bonding "mode" is set to "balance-xor" instead of "balance-rr": higher throughput with multiple connexions (no packet reordering)
- "xmit_hash_policy" is set to "layer3+4": TUN interface has no MAC address so balancing is done on upper level (source/dest IP + source/dest port)
- OpenVPN auth is disabled : less overhead, higher throughput
- OpenVPN cipher is set to "BF-CBC": AES-256-CBC consume too much CPU on client side (Atom CPU)
- Compression is always enabled (noadapt): higher upload speed when serving HTML/CSS/JS files

Tested (and approved) with an Atom (1.8 Ghz - client) and a Core I5 (3.3 Ghz - server).

Max download speed : 28 Mbit/s (I would expect 35 Mbit/s...)
Max upload speed : 1.8 Mbit/s (almost twice, this is good)

The main problem is the latency introduced: as far I know, throughput is highly dependent on latency so I think this is why I do not get a better download speed (I did not "play" with TCP windows size).

Just a question: is there an option or a way to tell OpenVPN to "group" multiple small packets inside a big one?

When downloading a file, there are many TCP ACK packets (40 bytes) sent.

But, problem is that with a PPPOE connexion, the overhead is huge : 40 bytes (IP) => 106 bytes (ATM, two cells).

If multiple ACK packets could be "encaspsulated" inside a "big" packet (MTU size), this would be much better.

Cheers!
Last edited by devnull on Thu Jan 23, 2014 8:45 pm, edited 1 time in total.

devnull
OpenVpn Newbie
Posts: 5
Joined: Thu Oct 24, 2013 5:29 pm

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by devnull » Thu Jan 23, 2014 8:43 pm

Sorry, bad edit posting...

User avatar
maikcat
Forum Team
Posts: 4200
Joined: Wed Jan 12, 2011 9:23 am
Location: Athens,Greece
Contact:

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by maikcat » Tue Jan 28, 2014 1:13 pm

thank you for sharing! :)

also keep in mind that :

layer3+4
This policy uses upper layer protocol information, when available, to generate the hash. This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves.
Regards,

Michael.
Amiga 500 , Zx +2 owner
Long live Dino Dini (Kick off 2 Creator)

Inflammable means flammable? (Dr Nick Riviera,Simsons Season13)

"objects in mirror are losing"

devnull
OpenVpn Newbie
Posts: 5
Joined: Thu Oct 24, 2013 5:29 pm

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by devnull » Thu Jan 30, 2014 10:51 pm

Hello! :)
maikcat wrote:thank you for sharing! :)
You are welcome!
maikcat wrote: although a single connection will not span multiple slaves.
That's right, a single connection won't be faster with balance-xor mode.

But, with multiple simultaneous connections, I noticed that throughput is much better (no packet reordering? I need to double-check that point)

And it is easier to aggregate different links even if they do not have the same throughput or latency.

Cheers!

josh_schamp
OpenVpn Newbie
Posts: 1
Joined: Wed Sep 02, 2015 2:55 pm

Re: Expert needed: OpenVPN bonding throughput less than expe

Post by josh_schamp » Wed Sep 02, 2015 3:00 pm

Hello, I know this thread is a bit old, however I was wondering if this same setup can be achieved using OpenVPN servers on different ports?

I have a single server in a datacenter, with one NIC and one IPv4 address.

I'd like to do this same setup and think I can set OpenVPN to bind to each WAN Interface locally and connect to separate ports on the server end.

q8reflex
OpenVpn Newbie
Posts: 1
Joined: Mon Feb 13, 2017 4:52 am

Re: Expert needed: OpenVPN bonding throughput less than expected

Post by q8reflex » Mon Feb 13, 2017 4:57 am

Ditto to that! appreciate if anyone can point or even make a guide to achieve easy-to-do openvpn bonding. Nowadays it would be easy to bond/aggregate several internet connections with a cheap cloud virtual sever.

ariusbb
OpenVpn Newbie
Posts: 1
Joined: Thu Apr 20, 2017 9:41 am

Re: Expert needed: OpenVPN bonding throughput less than expected

Post by ariusbb » Thu Apr 20, 2017 2:09 pm

Anyone has updated instructions for Ubuntu 16.04 LTS?
I want to bond two DSL lines to an Ubuntu server in a datacenter.
The best thing I could do was to start the bonding but performance was very slow (100-200kb/sec) and pings timed out.

Post Reply