Hello! (Please forgive my bad English) I have read OpenVPN manual three times, tested every option that could help but I am stuck... (Google did not help) At home, I have the luck to have two (very) good ADSL lines (120 ft from the "remote concentrator"). Each ADSL line (PPPOE) is stable and fast : 20 mbit/s IN 1 mbit/s OUT. A small Linux "box" (Atom 1.8 Ghz, 2 cores, HT) is acting as a router/firewall (Debian 7, lastest OpenVZ kernel) and traffic shaping is used (htb) to keep personnal VOIP clean. So, I have "bonded" the two ADSL lines with OpenVPN to a (fat) remote server (at work): 1 gbit/s NIC, throughput can stand 600 mbit/s IN or OUT without problem. OpenVPN Bonding is working well (mode 0, balance-rr) but download speed is "hitting a wall" : ~23 Mbit/s (while I could expect more than 35 Mbit/s). If I did not make any error, a MTU of 1448 bytes is perfect to exactly fit 30 ATM cells: 29 cells * 48 bytes + last cell (48 bytes + 8 bytes SAR trailer) = 1448 bytes. Tcpdump (-n -v -e -i vmbrX port 31415 and greater 1462 (1462 - 14 bytes for Ethernet)) output confirms this : [code]ethertype IPv4 (0x0800), length 1462: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1448) ip.addr.dest 31415 > ip.addr.source.34483: UDP, length 1420[/code] At first, I thought than the Atom processor was to slow to handle LZO compression, encryption and HMAC verification so I disabled everything: no TLS (static key), LZO (comp-lzo no), replay (no-replay), auth (none), cipher (none). That did not help. Now, (while downloading) I suspect ACK packets (inside the VPNs) to completly fill up the upload of each ADSL line but I also need to double-check with wireshark if the problem could be related to TCP DUP ACK/TCP Retransmission. Below are the configuration files in use. Any help on this problem would be greatly appreciated. :) Thanks! OpenVPN clients: [code] remote xxx.xxx.xxx.xxx 31415 proto udp nobind dev bond-client1 dev-type tun fragment 1420 mssfix sndbuf 262144 rcvbuf 262144 txqueuelen 256 ping-restart 0 persist-tun persist-key script-security 2 cd /home/master/root/etc/openvpn/bond-client-X/ log log/openvpn.log nice -20 fast-io verb 4 status log/status.log 300 comp-lzo no secret keys/static.key auth none cipher none no-replay up bin/up.sh down bin/down.sh [/code] OpenVPN servers: [code] local xxx.xxx.xxx.xxx proto udp port 31415 dev bond-server-X dev-type tun fragment 1420 mssfix sndbuf 262144 rcvbuf 262144 txqueuelen 256 ping-restart 0 persist-tun persist-key script-security 2 cd /home/master/root/etc/openvpn/bond-server-X/ log log/openvpn.log nice -20 fast-io verb 4 status log/status.log 300 comp-lzo no secret keys/static.key auth none cipher none no-replay up bin/up.sh down bin/down.sh [/code] bond0 interface (client & server) [code] iface bond0 inet static pre-up /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client-1 pre-up /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client-2 post-up /etc/init.d/openvpn start bond-client1 bond-client-2 pre-down /etc/init.d/openvpn stop bond-client1 bond-client-2 pre-down /bin/sleep 5 post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client-1 post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client-2 post-down /sbin/rmmod bonding bond_mode balance-rr slaves bond-client-1 bond-client-2 address 10.10.10.1 netmask 255.255.255.0 network 10.10.10.0 broadcast 10.10.10.255 [/code]

Expert needed: OpenVPN bonding throughput less than expected

devnull

Sorry, we encountered an error while displaying this content. If you're a user, please try again later. If you're an administrator, take a look in your Flarum log files for more information.

maikcat

are you sure your bonding setup works?

AFAIK you could only achieve bonding by using tap interface devices
which they operate at layer2, at least i have tested this and was working...

aa:~ # modinfo bonding
filename: /lib/modules/3.4.47-2.38-desktop/kernel/drivers/net/bonding/bonding.ko
alias: rtnl-link-bond
author: Thomas Davis, tadavis@lbl.gov and many others
description: Ethernet Channel Bonding Driver, v3.7.1

curious though, can you post also the contents of your up/down scripts?

Michael.

devnull

Many thanks for your answer! 🙂

I think I have found the problem...

=> I need to test, double-check, etc.

And when all is running smoothly, I will post every config files in this topic.

By the way, bonding over tun interfaces works!

devnull

Hello! 😃

So, problem (almost) solved, here we go:

Server config:

local <server_ip_address_1>
proto udp
port 1194
dev bond-server1
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
auth none
cipher BF-CBC
replay-window 256 60
tls-server
ca ca.crt
dh dh4096.pem
cert server-1.crt
key server-1.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 0

local <server_ip_address_2>
proto udp
port 1194
dev bond-server2
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
auth none
cipher BF-CBC
replay-window 256 60
tls-server
ca ca.crt
dh dh4096.pem
cert server-2.crt
key server-2.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 0

# auto bond0
iface bond0 inet static
        pre-up    /sbin/modprobe bonding
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-server1
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-server2
        post-up   /etc/init.d/openvpn start bond-server1 bond-server2
        pre-down  /etc/init.d/openvpn stop  bond-server1 bond-server2
        post-down /bin/sleep 5
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-server1
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-server2
        post-down /sbin/rmmod bonding
        bond_mode balance-xor
        bond_xmit_hash_policy layer3+4
        slaves bond-server1 bond-server2
        address 10.10.10.2
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255

Client config:

remote <server_ip_address_1> 1194
proto udp
nobind
dev bond-client1
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
client
auth none
cipher BF-CBC
replay-window 256 60
tls-client
ca ca.crt
cert client-1.crt
key client-1.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 1
ns-cert-type server

remote <server_ip_address_2> 1194
proto udp
nobind
dev bond-client2
dev-type tun
fragment 1420
mssfix
txqueuelen 1000
keepalive 10 30
persist-tun
persist-key
script-security 2
log-append openvpn.log
nice -20
fast-io
verb 4
status status.log 300
comp-lzo adaptive
comp-noadapt
client
auth none
cipher BF-CBC
replay-window 256 60
tls-client
ca ca.crt
cert client-2.crt
key client-2.key
key-method 2
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA
reneg-sec 900
tran-window 900
tls-auth ta.key 1
ns-cert-type server

# auto bond0
iface bond0 inet static
        pre-up    /sbin/modprobe bonding
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client1
        pre-up    /usr/sbin/openvpn --mktun --dev-type tun --dev bond-client2
        post-up   /etc/init.d/openvpn start bond-client1 bond-client2
        pre-down  /etc/init.d/openvpn stop  bond-client1 bond-client2
        post-down /bin/sleep 5
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client1
        post-down /usr/sbin/openvpn --rmtun --dev-type tun --dev bond-client2
        post-down /sbin/rmmod bonding
        bond_mode balance-xor
        bond_xmit_hash_policy layer3+4
        slaves bond-client1 bond-client2
        address 10.10.10.1
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255

Notes:

- Bonding "mode" is set to "balance-xor" instead of "balance-rr": higher throughput with multiple connexions (no packet reordering)
- "xmit_hash_policy" is set to "layer3+4": TUN interface has no MAC address so balancing is done on upper level (source/dest IP + source/dest port)
- OpenVPN auth is disabled : less overhead, higher throughput
- OpenVPN cipher is set to "BF-CBC": AES-256-CBC consume too much CPU on client side (Atom CPU)
- Compression is always enabled (noadapt): higher upload speed when serving HTML/CSS/JS files

Tested (and approved) with an Atom (1.8 Ghz - client) and a Core I5 (3.3 Ghz - server).

Max download speed : 28 Mbit/s (I would expect 35 Mbit/s...)
Max upload speed : 1.8 Mbit/s (almost twice, this is good)

The main problem is the latency introduced: as far I know, throughput is highly dependent on latency so I think this is why I do not get a better download speed (I did not "play" with TCP windows size).

Just a question: is there an option or a way to tell OpenVPN to "group" multiple small packets inside a big one?

When downloading a file, there are many TCP ACK packets (40 bytes) sent.

But, problem is that with a PPPOE connexion, the overhead is huge : 40 bytes (IP) => 106 bytes (ATM, two cells).

If multiple ACK packets could be "encaspsulated" inside a "big" packet (MTU size), this would be much better.

Cheers!

devnull

Sorry, bad edit posting...

maikcat

thank you for sharing! 🙂

also keep in mind that :

layer3+4
This policy uses upper layer protocol information, when available, to generate the hash. This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves.

Regards,

Michael.

devnull

Hello! 🙂

maikcat wrotethank you for sharing! 🙂

You are welcome!

maikcat wrote although a single connection will not span multiple slaves.

That's right, a single connection won't be faster with balance-xor mode.

But, with multiple simultaneous connections, I noticed that throughput is much better (no packet reordering? I need to double-check that point)

And it is easier to aggregate different links even if they do not have the same throughput or latency.

Cheers!

josh_schamp

Hello, I know this thread is a bit old, however I was wondering if this same setup can be achieved using OpenVPN servers on different ports?

I have a single server in a datacenter, with one NIC and one IPv4 address.

I'd like to do this same setup and think I can set OpenVPN to bind to each WAN Interface locally and connect to separate ports on the server end.

q8reflex

Ditto to that! appreciate if anyone can point or even make a guide to achieve easy-to-do openvpn bonding. Nowadays it would be easy to bond/aggregate several internet connections with a cheap cloud virtual sever.

ariusbb

Anyone has updated instructions for Ubuntu 16.04 LTS?
I want to bond two DSL lines to an Ubuntu server in a datacenter.
The best thing I could do was to start the bonding but performance was very slow (100-200kb/sec) and pings timed out.