Throughput Performance Issue

lbe

I recently upgraded my server on Linode from a 32-bit version of Fedora 17 to a 64-bit version of Fedora 20. When I migrated my OpenVPN configuration from the old machine to the new, I experienced a significant reduction of throughput for my vpn connections from well over 1 MBps to less than 20 KBps on average. I experienced the same reduction in throughput with pptpd connections to the same server. If I forward a local port using ssh to the server, then my throughput is as expected.

Functionally, both the OpenVPN and PPTP connections work - meaning that I can connect to everything that I want. The only issue is performance

I have performed multiple searches attempting to find any related issues and have not found any so far.

Thanks in advance for your assistance!

lbe

Test: Execute the follow wget command

wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip

When executed locally on the server, my response is

 whgi@localhost{~}:wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip
--2014-05-13 15:44:38--  http://speedtest.wdc01.softlayer.com/downloads/test10.zip
Resolving speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11536384 (11M) [application/zip]
Saving to: ‘/dev/null’

100%[======================================>] 11,536,384  13.7MB/s   in 0.8s

2014-05-13 15:44:39 (13.7 MB/s) - ‘/dev/null’ saved [11536384/11536384]

When I execute the same command from my office (which obviously has a slower internet connection ) without a vpn connection, I received the following:

alias st_wget='wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip'
 whgi@Williams-MacBook-Pro{~}:st_wget
--2014-05-13 10:46:23--  http://speedtest.wdc01.softlayer.com/downloads/test10.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11536384 (11M) [application/zip]
Saving to: '/dev/null'

100%[========================================================================>] 11,536,384   925KB/s   in 11s

2014-05-13 10:46:34 (1021 KB/s) - '/dev/null' saved [11536384/11536384]

The details of my configuration are:

OS: Fedora 20 (Linux localhost.localdomain 3.14.1-x86_64-linode39 #1 SMP Fri Apr 25 13:07:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux)

I am running OpenVPN using the package available with Fedora 20 - openvpn-2.3.2-4.fc20.x86_64. The output from openvpn --version is:

OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013
Originally developed by James Yonan
Copyright (C) 2002-2010 OpenVPN Technologies, Inc. <sales@openvpn.net>
Compile time defines: enable_crypto=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_eurephia=yes enable_fast_install=yes enable_fragment=yes enable_http_proxy=yes enable_iproute2=yes enable_libtool_lock=yes enable_lzo=yes enable_lzo_stub=no enable_management=yes enable_multi=yes enable_multihome=yes enable_pam_dlopen=no enable_password_save=yes enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_pthread=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_socks=yes enable_ssl=yes enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=no enable_win32_dll=yes enable_x509_alt_username=yes with_crypto_library=openssl with_gnu_ld=yes with_iproute_path=/sbin/ip with_mem_check=no with_plugindir='$(libdir)/openvpn/plugins' with_sysroot=no

My server configuration file is:

port 1194
dev tun

tls-server
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/host1.crt
key /etc/openvpn/keys/host1.key
dh /etc/openvpn/keys/dh1024.pem

mode server

ifconfig 10.8.0.1 10.8.0.2

ifconfig-pool 10.8.0.9 10.8.0.28

push "route 10.8.0.1 255.255.255.255"

push "dhcp-option DOMAIN mydomain.com"
push "dhcp-option DNS 4.2.2.1"

keepalive 10 60

inactive 600

route 10.8.0.0 255.255.255.0

user openvpn
group openvpn

persist-tun
persist-key

status     /var/log/openvpn/openvpn-status.log
log-append /var/log/openvpn/openvpn.log
verb 4

push "redirect-gateway def1" # bypass-dhcp"
comp-lzo

My client configuration file is:

dev tun

proto udp

remote host1.mydomain.com 1194

resolv-retry infinite

nobind

persist-key
persist-tun

ca ca.crt
cert client.crt
key client.key

comp-lzo

verb 3

Results of an ifconfig -a command (IP addresses anonymized)

dummy0: flags=130<BROADCAST,NOARP>  mtu 1500
        ether 3a:59:00:d2:d4:7d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XX.XX.XXX.XX  netmask 255.255.255.0  broadcast 70.85.129.255
        ether fe:fd:46:55:81:3b  txqueuelen 1000  (Ethernet)
        RX packets 67921  bytes 100790033 (96.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55018  bytes 25013673 (23.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

gre0: flags=128<NOARP>  mtu 1476
        unspec 00-00-00-00-00-00-F0-10-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

gretap0: flags=4098<BROADCAST,MULTICAST>  mtu 1476
        ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip6_vti0: flags=128<NOARP>  mtu 1500
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip6gre0: flags=128<NOARP>  mtu 1448
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip6tnl0: flags=128<NOARP>  mtu 1452
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip_vti0: flags=128<NOARP>  mtu 1500
        tunnel   txqueuelen 0  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 0  (Local Loopback)
        RX packets 710  bytes 79408 (77.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 710  bytes 79408 (77.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

sit0: flags=128<NOARP>  mtu 1480
        sit  txqueuelen 0  (IPv6-in-IPv4)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

teql0: flags=128<NOARP>  mtu 1500
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 10.8.0.1  netmask 255.255.255.255  destination 10.8.0.2
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 1774  bytes 182321 (178.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1698  bytes 1156086 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunl0: flags=128<NOARP>  mtu 0
        tunnel   txqueuelen 0  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And last but not least, my firewall script. I normal use the ARNO firewall rules script. However, it has a much more complicated rule set and I wanted to simplify my rules for the purpose of this performance test.

#!/bin/sh

# A Sample OpenVPN-aware firewall.

# eth0 is connected to the internet.

# Change this subnet to correspond to your private
# ethernet subnet.  Home will use HOME_NET/24 and
# Office will use OFFICE_NET/24.
PRIVATE=10.0.0.0/24

# Loopback address
LOOP=127.0.0.1

# Delete old iptables rules
# and temporarily block all traffic.
iptables -P OUTPUT DROP
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -F

# Set default policies
iptables -P OUTPUT ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP

# Prevent external packets from using loopback addr
iptables -A INPUT -i eth0 -s $LOOP -j DROP
iptables -A FORWARD -i eth0 -s $LOOP -j DROP
iptables -A INPUT -i eth0 -d $LOOP -j DROP
iptables -A FORWARD -i eth0 -d $LOOP -j DROP

# Anything coming from the Internet should have a real Internet address
iptables -A FORWARD -i eth0 -s 192.168.0.0/16 -j DROP
iptables -A FORWARD -i eth0 -s 172.16.0.0/12 -j DROP
iptables -A FORWARD -i eth0 -s 10.0.0.0/8 -j DROP
iptables -A INPUT -i eth0 -s 192.168.0.0/16 -j DROP
iptables -A INPUT -i eth0 -s 172.16.0.0/12 -j DROP
iptables -A INPUT -i eth0 -s 10.0.0.0/8 -j DROP

# Block outgoing NetBios (if you have windows machines running
# on the private subnet).  This will not affect any NetBios
# traffic that flows over the VPN tunnel, but it will stop
# local windows machines from broadcasting themselves to
# the internet.
iptables -A FORWARD -p tcp --sport 137:139 -o eth0 -j DROP
iptables -A FORWARD -p udp --sport 137:139 -o eth0 -j DROP
iptables -A OUTPUT -p tcp --sport 137:139 -o eth0 -j DROP
iptables -A OUTPUT -p udp --sport 137:139 -o eth0 -j DROP

# Check source address validity on packets going out to internet
#iptables -A FORWARD -s ! $PRIVATE -i eth1 -j DROP

# Allow local loopback
iptables -A INPUT -s $LOOP -j ACCEPT
iptables -A INPUT -d $LOOP -j ACCEPT

# Allow incoming pings (can be disabled)
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Allow services such as www and ssh (can be disabled)
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 1723 -j ACCEPT
iptables -A INPUT -p tcp --dport 6013 -j ACCEPT
iptables -A INPUT -p tcp --dport 10000 -j ACCEPT
iptables -A INPUT -p gre -j ACCEPT

# Allow incoming OpenVPN packets
# Duplicate the line below for each
# OpenVPN tunnel, changing --dport n
# to match the OpenVPN UDP port.
#
# In OpenVPN, the port number is
# controlled by the --port n option.
# If you put this option in the config
# file, you can remove the leading '--'
#
# If you taking the stateful firewall
# approach (see the OpenVPN HOWTO),
# then comment out the line below.

iptables -A INPUT -p udp --dport 1194 -j ACCEPT
iptables -A INPUT -p udp --dport 53 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow packets from TUN/TAP devices.
# When OpenVPN is run in a secure mode,
# it will authenticate packets prior
# to their arriving on a tun or tap
# interface.  Therefore, it is not
# necessary to add any filters here,
# unless you want to restrict the
# type of packets which can flow over
# the tunnel.

iptables -A INPUT -i tun+ -j ACCEPT
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A INPUT -i tap+ -j ACCEPT
iptables -A FORWARD -i tap+ -j ACCEPT
iptables -A INPUT -i ppp+ -j ACCEPT
iptables -A FORWARD -i ppp+ -j ACCEPT


# Keep state of connections from local machine and private subnets
iptables -A OUTPUT -m state --state NEW -o eth0 -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state NEW -o eth0 -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Masquerade local subnet used for OpenVPN
iptables -t nat -A POSTROUTING -s $PRIVATE -o eth0 -j MASQUERADE
#Masquerade local subnet used for pptpd
iptables -t nat -A POSTROUTING -s 192.168.0.0/16 -o eth0 -j MASQUERADE

lbe

Debbie, I apologize but I am confused. While I am happy to help out the project with provide test data in a format that it desires, I don't think the project would benefit from having test data on a less than perfomant system that I probably have misconfigured. I am posting asking for assistance in identifying what I have misconfigured. Prior to make the move to 64-bit Fedora 20, My throughput would easily reach 7 - 8 Mbps. This was on a much slower physical server hosting my virtual server. I am now on a much faster physical server and everything "but" OpenVPN and pptpd are running much faster. These two are running two to three orders of magnitude slower. I have little doubt that I have something screwed up, but I can't find it given my current knowledge.

I am hoping that someone reading this will have had a similar situation and can point me in the right direction. Since the main commonality between OpenVPN and pptpd if the forwarding of packets and masquerading, I am guessing that my problem is associated with one of these two dimensions. I have stripped my iptables configuration to pretty much the absolute minimum and have not seen any performance improvements

Are there settings for tun+ and ppp+ devices that I am missing that would increase their performance?

Again, thanks in advance to all who read this and especially to those who respond with guidance.

lbe

lbe

Thanks to the superb technical support staff at Linode, I have resolved my performance problem.

There is a problem with kernel that I was running:

Linux localhost.localdomain 3.14.1-x86_64-linode39 #1 SMP Fri Apr 25 13:07:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

As suggested by the Linode technical support staff, I bumped my kernel version back to:

Linux localhost.localdomain 3.10.3-x86_64-linode34 #1 SMP Fri Jul 26 13:21:11 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

Now my performance is back to normal

--2014-05-15 17:19:33--  http://speedtest.wdc01.softlayer.com/downloads/test10.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11536384 (11M) [application/zip]
Saving to: '/dev/null'

100%[======================================================================>] 11,536,384  1.02MB/s   in 12s

2014-05-15 17:19:45 (961 KB/s) - '/dev/null' saved [11536384/11536384]

cron2

This is indeed a known problem in 3.14 and up - they have changed quite a bit of stuff and one of the changes broke througput through the tun/tap driver. It is know, a patch has been posted, but I'm not sure when distributions will pick it up.

See here:

https://bugzilla.kernel.org/show_bug.cgi?id=73891
https://bugzilla.kernel.org/show_bug.cgi?id=74051

gert