Speed issue .. maybe mtu issue

alexs_yb

server conf

udp

local 10.172.203.29
server 10.172.210.0 255.255.255.128
server-ipv6 2002:ca4a:2000:ACD2:4000::/66
port 2443
proto udp
dev tun
topology subnet
ifconfig-pool-persist /etc/openvpn/ip-pool.txt
txqueuelen 500
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key  # This file should be kept secret
dh /etc/openvpn/dh2048.pem
crl-verify /etc/openvpn/client-crl.pem
tls-verify /etc/openvpn/verifyCert.sh
tls-auth /etc/openvpn/ta.key 0 # This file is secret
client-config-dir ccd
push "route-ipv6 2001::/16"
push "route-ipv6 2002::/16"
push "route-ipv6 0::/1"
push "route-ipv6 8000::/1"
push "redirect-gateway def1"
push "dhcp-option DNS 10.32.20.102"
push "dhcp-option DNS 10.32.20.100"
push "dhcp-option DNS 10.32.69.11"
push "comp-lzo yes"
keepalive 15 45
cipher BF-CBC        # Blowfish (default)
comp-lzo
max-clients 100
user nobody
group nobody
persist-key
persist-tun
status /etc/openvpn/status-udp.log
verb 3
mute 10
management localhost 2443
script-security 2
client-connect /etc/openvpn/ConnectClientFW.sh
client-disconnect  /etc/openvpn/DisConnectClientFW.sh
learn-address /etc/openvpn/set-route.sh

tcp

local 10.172.203.29
server 10.172.210.0 255.255.255.128
server-ipv6 2002:ca4a:2000:ACD2:0000::/66
port 443
proto tcp
dev tun
topology subnet
ifconfig-pool-persist /etc/openvpn/ip-pool.txt
txqueuelen 500
tcp-queue-limit 500
tcp-nodelay
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key  # This file should be kept secret
dh /etc/openvpn/dh2048.pem
crl-verify /etc/openvpn/client-crl.pem
tls-verify /etc/openvpn/verifyCert.sh
tls-auth /etc/openvpn/ta.key 0 # This file is secret
client-config-dir ccd
push "route-ipv6 2001::/16"
push "route-ipv6 2002::/16"
push "route-ipv6 0::/1"
push "route-ipv6 8000::/1"
push "redirect-gateway def1"
push "dhcp-option DNS 10.32.20.100"
push "dhcp-option DNS 10.32.20.102"
push "dhcp-option DNS 10.32.69.11"
push "comp-lzo yes"
keepalive 15 45
cipher BF-CBC        # Blowfish (default)
comp-lzo
max-clients 100
user nobody
group nobody
persist-key
persist-tun
status /etc/openvpn/status-tcp.log
verb 3
mute 10
management localhost 443
script-security 2
client-connect /etc/openvpn/ConnectClientFW.sh
client-disconnect  /etc/openvpn/DisConnectClientFW.sh
learn-address /etc/openvpn/set-route.sh

client 
dev tun
resolv-retry infinite
nobind
remote-cert-tls server
tls-client
ca client-ca.crt
tls-auth ta.key 1
cryptoapicert "SUBJ: vpn"
verify-x509-name 'C=AU,CN=xxxxx'

comp-lzo
verb 3
keepalive 5 15
auth-retry nointeract 

socket-flags TCP_NODELAY
<connection>
remote vpn.abc.com 2443 udp
</connection>

# Try TCP next but max of 5 times
<connection>
# retry every 1 sec
connect-retry 1
# time out every 10s
connect-timeout 10
# keep try forever
#connect-retry-max infinite
connect-retry-max 5
remote vpn.yieldbroker.com 443 tcp
# only try this for 5 times
</connection>

When I have a W7 or W8 or W8.1 client behind this openvpn setup

with the server running on this centos 6.6 box
openvpn-2.3.7-1.el6.x86_64
Linux yboinfinfr6101 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I get overly large size packets.

I had staff complaining about that the vpn was slow, even unusable. so I set up a 100M.iso file to test downloading. it is hosted on a samba centos box.

I can scp / rsync it outside the vpn at high speed. when i try and to it from a window client I get about 30-40% of the way through, then it stops and waits awhile and then starts up and then stops etc etc. Seems like large transfers get stuck.

I put tcpdump on the openvpn server linux box and I can see the tcp mss is set to approx 1360 from memory but then I see larger packet being sent, larger then 1500. I also see icmp's being returned to say fragment but that doesn't seem to help.

I moved the file to a W2k8r2 box and the same result.

I found this article about a kernel issue, but i dont think this is the case here. same symptoms

I setup a test box linux to run a trial

test code

#!/bin/sh


/etc/init.d/openvpn start > /dev/null 2> /dev/null
sleep 5
ip r r 10.172.203.34/32 via 10.172.210.1
rm -f 100M.iso
echo "Start test over vpn"
rsync  -h -av --progress ybonas:/share/MD0_DATA/Downloads/temp/alex/vpn/201506/100M.iso .
/etc/init.d/openvpn stop > /dev/null 2> /dev/null
sleep 5
rm -f 100M.iso
echo "Start test outside vpn"
rsync  -h -av --progress ybonas:/share/MD0_DATA/Downloads/temp/alex/vpn/201506/100M.iso .
rm -f 100M.iso

This is ip 10.172.203.34 is ybonas ..

you can see the difference

Start test over vpn
receiving incremental file list
100M.iso
     104.86M 100%    1.25MB/s    0:01:19 (xfer#1, to-check=0/1)

sent 30 bytes  received 104.87M bytes  1.30M bytes/sec
total size is 104.86M  speedup is 1.00
Start test outside vpn
receiving incremental file list
100M.iso
     104.86M 100%   87.95MB/s    0:00:01 (xfer#1, to-check=0/1)

sent 30 bytes  received 104.87M bytes  41.95M bytes/sec
total size is 104.86M  speedup is 1.00

a small bit of the tcp dump

22:50:02.830599 IP 10.172.203.34.ssh > 10.172.210.30.37361: Flags [.], seq 92156702:92158058, ack 40814, win 41, options [nop,nop,TS val 3941403462 ecr 3941781807], length 1356
22:50:02.831731 IP 10.172.210.30.37361 > 10.172.203.34.ssh: Flags [.], ack 92158058, win 3461, options [nop,nop,TS val 3941782011 ecr 3941403462], length 0
22:50:02.831769 IP 10.172.203.34.ssh > 10.172.210.30.37361: Flags [.], seq 92227214:92229926, ack 40814, win 41, options [nop,nop,TS val 3941403463 ecr 3941782011], length 2712
22:50:02.832278 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556

this is similar to the samba issue.

alexs_yb

I have added in the syn packets at the beginning ..

23:04:12.389606 IP 10.172.210.30.42872 > 10.172.203.34.ssh: Flags [S], seq 975535236, win 14480, options [mss 1368,nop,nop,TS val 3942631568 ecr 3942242836,nop,wscale 10], length 0
23:04:12.389633 IP 10.172.203.34.ssh > 10.172.210.30.42872: Flags [S.], seq 2004319178, ack 975535237, win 14480, options [mss 1460,nop,nop,TS val 3942253021 ecr 3942631568,nop,wscale 10], length 0

1368 over opevpn ..

I found this
http://serverfault.com/questions/459844/server-sending-out-packets-bigger-then-mtu

I am running these boxes in VMWare ESX 5.5i with vmxnet3 nic's

alexs_yb

also
http://rtoodtoo.net/generic_segmentation_offload_and_wireshark/

i have turned off gso tso on ybonas and I see this

strange I am getting icmp, but it looks like outbound doesn't have larger mtu packets

23:17:33.927282 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556
23:17:34.127656 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39984650:39986006, ack 17726, win 32, options [nop,nop,TS val 3943054759 ecr 3943433105], length 1356
23:17:34.128691 IP 10.172.210.30.47534 > 10.172.203.34.ssh: Flags [.], ack 39986006, win 3306, options [nop,nop,TS val 3943433307 ecr 3943054759], length 0
23:17:34.128712 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39986006:39987362, ack 17726, win 32, options [nop,nop,TS val 3943054759 ecr 3943433307], length 1356
23:17:34.128729 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39987362:39988718, ack 17726, win 32, options [nop,nop,TS val 3943054759 ecr 3943433307], length 1356
23:17:34.129071 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556
23:17:34.328640 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39986006:39987362, ack 17726, win 32, options [nop,nop,TS val 3943054960 ecr 3943433307], length 1356
23:17:34.329747 IP 10.172.210.30.47534 > 10.172.203.34.ssh: Flags [.], ack 39987362, win 3306, options [nop,nop,TS val 3943433508 ecr 3943054960], length 0
23:17:34.329773 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39987362:39988718, ack 17726, win 32, options [nop,nop,TS val 3943054960 ecr 3943433508], length 1356
23:17:34.329792 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39988718:39990074, ack 17726, win 32, options [nop,nop,TS val 3943054960 ecr 3943433508], length 1356
23:17:34.330145 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556
23:17:34.529649 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39987362:39988718, ack 17726, win 32, options [nop,nop,TS val 3943055161 ecr 3943433508], length 1356
23:17:34.530663 IP 10.172.210.30.47534 > 10.172.203.34.ssh: Flags [.], ack 39988718, win 3306, options [nop,nop,TS val 3943433709 ecr 3943055161], length 0
23:17:34.530688 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39988718:39990074, ack 17726, win 32, options [nop,nop,TS val 3943055161 ecr 3943433709], length 1356
23:17:34.530708 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39990074:39991430, ack 17726, win 32, options [nop,nop,TS val 3943055161 ecr 3943433709], length 1356
23:17:34.531036 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556

this is what i see on the openvpn server ..... suddenly larger mtu packets

23:17:26.064157 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556
23:17:26.264908 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39930410:39931766, ack 17678, win 32, options [nop,nop,TS val 3943046897 ecr 3943425243], length 1356
23:17:26.265719 IP 10.172.210.30.47534 > 10.172.203.34.ssh: Flags [.], ack 39931766, win 3306, options [nop,nop,TS val 3943425445 ecr 3943046897], length 0
23:17:26.266120 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39931766:39934478, ack 17678, win 32, options [nop,nop,TS val 3943046898 ecr 3943425445], length 2712
23:17:26.266146 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556
23:17:26.466875 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39931766:39933122, ack 17678, win 32, options [nop,nop,TS val 3943047099 ecr 3943425445], length 1356
23:17:26.467654 IP 10.172.210.30.47534 > 10.172.203.34.ssh: Flags [.], ack 39933122, win 3306, options [nop,nop,TS val 3943425647 ecr 3943047099], length 0
23:17:26.468061 IP 10.172.203.34.ssh > 10.172.210.30.47534: Flags [.], seq 39933122:39935834, ack 17678, win 32, options [nop,nop,TS val 3943047100 ecr 3943425647], length 2712
23:17:26.468081 IP 10.172.203.30 > 10.172.203.34: ICMP 10.172.210.30 unreachable - need to frag (mtu 1500), length 556

I turned off all the offloading on the openvpn server nic. icmp ... fragment icmps stop.. yeah

but I still have a speed cap

Start test over vpn
receiving incremental file list
100M.iso
104.86M 100% 10.93MB/s 0:00:09 (xfer#1, to-check=0/1)

sent 30 bytes received 104.87M bytes 11.04M bytes/sec
total size is 104.86M speedup is 1.00
Start test outside vpn
receiving incremental file list
100M.iso
104.86M 100% 88.81MB/s 0:00:01 (xfer#1, to-check=0/1)

sent 30 bytes received 104.87M bytes 41.95M bytes/sec
total size is 104.86M speedup is 1.00

Traffic

alexs_yb wrotebut I still have a speed cap

You use both UDP and TCP in your setup and you have not made it clear which VPN you are testing.

FWIW: UDP is faster than TCP for OpenVPN.

alexs_yb

Hi

Yes sorry, udp by default is the one tested and preferred.

tcp is setup because we have always on vpn setup.

Traffic

You "speed issues" look normal to me.

Have you read this:
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux