tcp dup ack from wireshark, is this a problem?

mmruzik

Hello, I've been working a bit with openvpn here, and was wondering a few things.

While using wireshark on the TUN device to try diagnosing some performance issues between sites, I noticed a semi-common occurrence of TCP retransmission ack and dup packets on the tun interface. While communication is now fine, it was previously giving some delays, inconsistent pings, sometimes large pings. Traffic has since seemed to drop off on the interface, which is likely unrelated.

The issue is I'm not sure if these packets are fine to ignore or signs of network settings not optimized, or possibly a problem. I did try working with MTU sizes some, using tun-mtu, mssfix, and fragment to try getting a different result, but with no difference. I still get these retransmission and dup packets, sometimes a few at a time, sometimes one and not the other, sometimes a good block of 10 or more.

Could this be a known bug or just a latency issue? the ethx port doesn't see any of these problems, there doesn't seem to be a problem with memory or with cpu power on the two servers (multicore with plenty of cpu time available). Ping times between sites is usually around 20ms. The window size doesn't seem to shrink noticeably in the logs.

I experimented with ping sizes to see that mtu tops out around 1472, but even with that set, my test rig had the very same messages, it makes me think that by the nature of vpn, there will be some packets delayed a bit longer than normal.

Technically, nothing is complaining, or erroring out. I'm just worried that with the increased demands on bandwidth, I'm losing some to duplicates and retransmissions.

Server conf

dev tun0
server x.y.z.0 255.255.255.0
ifconfig-pool-persist ipp.txt
client-config-dir omitted
status openvpn-status.log
tls-server

ca omitted
cert omitted
key omitted
dh omitted
client-to-client
port 1194

user nobody
group nobody
comp-lzo

persist-key
persist-tun
ping-timer-rem
keepalive 5 45
;keepalive 10 20

verb 1
route wanlink1.0 255.255.255.0 x.y.z.1
route wanlink2.0 255.255.255.0 x.y.z.1
route wanlink3.0 255.255.255.0 x.y.z.1

remote office conf

dev tun
remote x.y.z.0
port 1194
pull
tls-client

ping 15
ping-restart 45
ping-timer-rem

persist-tun
persist-key
user nobody
group nobody
comp-lzo

tls-remote omitted
ca omitted
cert omitted
key omitted
status openvpn-status.log

Test rig wireshark output

4363	52.819707	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#8] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=655708 SLE=642916 SRE=644499	64386
4364	52.822357	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#9] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=656571 SLE=642916 SRE=644499	64386
4365	52.828198	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#10] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=657885 SLE=642916 SRE=644499	64386
4366	52.830788	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#11] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=657911 SLE=642916 SRE=644499	64386
4367	52.833845	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#12] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=659225 SLE=642916 SRE=644499	64386
4368	52.833893	10.10.0.254	10.8.0.13	TCP	[TCP Retransmission] ideafarm-door > m-wnn [ACK] Seq=645788 Ack=1182 Win=65535 Len=1314	65535
4369	52.836375	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#13] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=659481 SLE=642916 SRE=644499	64386
4370	52.841776	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#14] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=660795 SLE=642916 SRE=644499	64386
4371	52.841823	10.10.0.254	10.8.0.13	TCP	[TCP Retransmission] ideafarm-door > m-wnn [ACK] Seq=647102 Ack=1182 Win=65535 Len=43	65535
4372	52.844916	10.8.0.13	10.10.0.254	TCP	[TCP Dup ACK 4349#15] m-wnn > ideafarm-door [ACK] Seq=1182 Ack=642891 Win=64386 Len=0 SLE=647145 SRE=661283 SLE=642916 SRE=644499	64386

Any hints would be greatly appreciated.

janjust

your client/server config files don't mention any line 'proto udp' or 'proto tcp' so I assume the VPN transport protocol is UDP. Using TCP should be avoided anyways, so this is good 🙂

So there are retransmits occurring inside the tunnel - that could hint at some udp traffic loss of the tunnel itself. there's very little that can be done about that. Check the server and client log files, possibly with 'verb 7' set, to see if there is a large amount of packet loss.

mmruzik

Thanks! yeah, its UDP. The funny thing is that with the added verbage on the log, on my test rig, wireshark reported TCP window full suddenly instead of so many dup acks. The tricky part is that I thought that the window size was managed automatically in linux.

The server is Centos 5, the client was a windows 7 work laptop I had handy. I'll be able to put the changes into the production server to get more data soon, for now I'm working with the test rig.

mulderlr

I have this problem with OpenVPN 2.2.2 running as a server on CentOS 6.2 or 6.3 x86_64. Basically VPN clients (windows or linux) connect fine, but after TCP packets start flowing, ICMP fragment needed and TCP retransmits basically make the VPN unusable. Using the exact same configs on mandriva 2010.2 x86_64 server work fine. Anyone interested shedding some light? I can post detailed configs including wireshark captures if necessary. I have tried building and compiling on CentOS from tar.gz and using dag weirs rpms. I have tried rolling back versions to 2.1 or 2.0 openvpn on CentOS with no change in behavior. It seems to be a CentOS or linux kernel issue on that distribution to me.

mulderlr

forgot to mention that this is under vmware ESXI 5.1 and on CentOS using the vmxnet3 adapters with VMwareTools-8.6.5-731933 or VMwareTools-9.0.0-782409 installed. Found a link to this article http://www.ryanfrantz.com/2011/02/03/tcp-segmentation-offload/ which gave me the idea to play around with TSO settings on the vmxnet3 adapters. By repeatedly switching on all offload options (sg tso gso gro lro) and then switching them off again e.g. (ethtool -K ethX tso off) all of a sudden the problem is gone. So something must be goofy in vmxnet3 on linux when it comes to offload settings.