Windows 7 64-Bit tcp mode high latency

[huebeni]

Hi,

i am using OpenVPN client to connect to my company's VPN-Gateway. The client is running Windows 7 in 64-Bit mode.
The gateway is configured in "proto tcp" mode. The VPN works, but is terribly slow. I have ping >500ms to internal servers, though pings times to the gateway are about 10-20ms.

I tried the same client setup on a Windows XP 32-Bit machine. Same network. There it works like charm with ping times around 20ms.

Client Software:
OpenVPN 2.1_rc22
TAP-Driver 32-Bit v9 (tap0901.sys)

I read various problem reports like this, but no real solution so far.

Is there an solution available? I can't change the server-setup or can't reinstall windows.

Best regards,
Michael

[maikcat]

hi there,

please use openvpn v2.1.4 for win7...

cheers,

Michael.

[janjust]

latest version of openvpn for Windows 7 is 2.1.4 ; try upgrading to that. You can also try 2.2RC , which has an updated driver for Windows 7 as well.

The thing to try is to set the 'tcp-nodelay' flag on the server side, but as you don't have control over it ....

Please post your client configuration and connection log with 'verb 5' enabled - perhaps there's something odd .

[huebeni]

Hi,

thank you for your responses!
I upgraded the VPN Client to the 2.2RC, but nothing changed in VPN performance

Now I'll include my current configuration and log output (obfuscated ips and hostnames).
I want to stress, that this configuration works fast and fine with Windows XP.

Bye,
Michael

Code: Select all

ip-win32 dynamic
client
dev tun
proto tcp
remote portal.some-domain.com 443
tls-remote "<certificate information>"
resolv-retry infinite
nobind
persist-key
persist-tun
ca <certificate information>
cert <certificate information>
key <certificate information>
auth-user-pass
cipher AES-256-CBC
auth SHA1
comp-lzo
route-delay 4
verb 3
reneg-sec 0

Code: Select all

Thu Apr 21 09:42:55 2011 OpenVPN 2.2-RC2 Win32-MSVC++ [SSL] [LZO2] built on Mar 25 2011
Thu Apr 21 09:43:02 2011 WARNING: Make sure you understand the semantics of --tls-remote before using it (see the man page).
Thu Apr 21 09:43:02 2011 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Thu Apr 21 09:43:02 2011 LZO compression initialized
Thu Apr 21 09:43:02 2011 Control Channel MTU parms [ L:1560 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Apr 21 09:43:02 2011 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Apr 21 09:43:02 2011 Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Apr 21 09:43:02 2011 Local Options hash (VER=V4): '958c5492'
Thu Apr 21 09:43:02 2011 Expected Remote Options hash (VER=V4): '79ef4284'
Thu Apr 21 09:43:02 2011 Attempting to establish TCP connection with 1.2.3.134:443
Thu Apr 21 09:43:02 2011 TCP connection established with 1.2.3.134:443
Thu Apr 21 09:43:02 2011 TCPv4_CLIENT link local: [undef]
Thu Apr 21 09:43:02 2011 TCPv4_CLIENT link remote: 1.2.3.134:443
Thu Apr 21 09:43:03 2011 TLS: Initial packet from 1.2.3.134:443, sid=f9bdbcd6 d3c5955b
Thu Apr 21 09:43:03 2011 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Thu Apr 21 09:43:07 2011 VERIFY OK: depth=1, <certificate information>
Thu Apr 21 09:43:07 2011 VERIFY X509NAME OK: <certificate information>
Thu Apr 21 09:43:07 2011 VERIFY OK: depth=0, <certificate information>
Thu Apr 21 09:43:14 2011 Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Thu Apr 21 09:43:14 2011 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 21 09:43:14 2011 Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Thu Apr 21 09:43:14 2011 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 21 09:43:14 2011 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Thu Apr 21 09:43:14 2011 [some.host.com] Peer Connection Initiated with 1.2.3.134:443
Thu Apr 21 09:43:17 2011 SENT CONTROL [some.host.com]: 'PUSH_REQUEST' (status=1)
Thu Apr 21 09:43:18 2011 PUSH: Received control message: 'PUSH_REPLY,ifconfig 10.x.x.250 10.x.x.249,ping-restart 120,ping 10,topology net30,route 10.x.x.1,dhcp-option DOMAIN some domain,dhcp-option WINS 10.x.x.252,dhcp-option DNS 10.x.x.254,route <some routes>'
Thu Apr 21 09:43:18 2011 OPTIONS IMPORT: timers and/or timeouts modified
Thu Apr 21 09:43:18 2011 OPTIONS IMPORT: --ifconfig/up options modified
Thu Apr 21 09:43:18 2011 OPTIONS IMPORT: route options modified
Thu Apr 21 09:43:18 2011 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Thu Apr 21 09:43:18 2011 ROUTE default_gateway=192.168.99.1
Thu Apr 21 09:43:18 2011 TAP-WIN32 device [Local Area Connection 2] opened: \\.\Global\{F8EE3EE1-F67C-4BFD-BD03-EB027FD460EB}.tap
Thu Apr 21 09:43:18 2011 TAP-Win32 Driver Version 9.8 
Thu Apr 21 09:43:18 2011 TAP-Win32 MTU=1500
Thu Apr 21 09:43:18 2011 Notified TAP-Win32 driver to set a DHCP IP/netmask of 10.x.x.250/255.255.255.252 on interface {F8EE3EE1-F67C-4BFD-BD03-EB027FD460EB} [DHCP-serv: 10.x.x.249, lease-time: 31536000]
Thu Apr 21 09:43:18 2011 Successful ARP Flush on interface [55] {F8EE3EE1-F67C-4BFD-BD03-EB027FD460EB}
Thu Apr 21 09:43:22 2011 TEST ROUTES: 12/12 succeeded len=12 ret=1 a=0 u/d=up
Thu Apr 21 09:43:22 2011 C:\WINDOWS\system32\route.exe ADD 10.x.x.1 MASK 255.255.255.255 10.x.x.249
Thu Apr 21 09:43:22 2011 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=30 and dwForwardType=4
Thu Apr 21 09:43:22 2011 Route addition via IPAPI succeeded [adaptive]
<more routes>
Thu Apr 21 09:43:22 2011 C:\WINDOWS\system32\route.exe ADD 172.x.x.0 MASK 255.255.255.0 10.x.x.249
Thu Apr 21 09:43:22 2011 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=30 and dwForwardType=4
Thu Apr 21 09:43:22 2011 Route addition via IPAPI succeeded [adaptive]
Thu Apr 21 09:43:22 2011 Initialization Sequence Completed

[janjust]

very strange... the connection log shows no errors;
can you try adding
mssfix 0
to see if it has an effect?

[huebeni]

Hi,

thanks for the mssfix reply, but it didn't improve the network performance.

I did some more analysis. I traced packets during pinging an internal server on the TAP and on the external interface.
On the TAP (tunnel) interface I can see the ping and ping-reply packets.
Time Ping-Request seen on TAP to external interface out: <1ms
Time for encrypted reply to come from gateway: 12ms
Time for packet to be decoded an appearing on the internal TAP interface: 507ms
The replys come in with an delay of about 520ms.

So the path out through OpenVPN, the gateway, private network, gateway, external interface seems to be ok and working.
But something goes wrong with the incoming packet until it appears on the internal interface. It takes about 500ms to decode???

Here comes the OpenVPN log output for two such cases:

Code: Select all

Tue Apr 26 11:17:21 2011 us=320000 TUN READ [60]
Tue Apr 26 11:17:21 2011 us=320000 TLS: tls_pre_encrypt: key_id=0
Tue Apr 26 11:17:21 2011 us=320000 TCPv4_CLIENT WRITE [117] to 1.2.3.134:443: P_DATA_V1 kid=0 DATA len=116
Tue Apr 26 11:17:21 2011 us=835000 TCPv4_CLIENT READ [117] from 1.2.3.134:443: P_DATA_V1 kid=0 DATA len=116
Tue Apr 26 11:17:21 2011 us=835000 TLS: tls_pre_decrypt, key_id=0, IP=1.2.3.134:443
Tue Apr 26 11:17:21 2011 us=835000 TUN WRITE [60]
Tue Apr 26 11:17:22 2011 us=334000 TUN READ [60]
Tue Apr 26 11:17:22 2011 us=334000 TLS: tls_pre_encrypt: key_id=0
Tue Apr 26 11:17:22 2011 us=334000 TCPv4_CLIENT WRITE [117] to 1.2.3.134:443: P_DATA_V1 kid=0 DATA len=116
Tue Apr 26 11:17:22 2011 us=849000 TCPv4_CLIENT READ [117] from 1.2.3.134:443: P_DATA_V1 kid=0 DATA len=116
Tue Apr 26 11:17:22 2011 us=849000 TLS: tls_pre_decrypt, key_id=0, IP=1.2.3.134:443
Tue Apr 26 11:17:22 2011 us=849000 TUN WRITE [60]

Here you can see the read is about 500ms after the write. But the packet has arrived a lot earlier.

Maybe this information helps to improve this issue.

Best regards,
Michael

[janjust]

that's a very interesting discovery!
so there seems to be a 500 ms delay between the moment the packet is written out to the 'tap-win32' interface and the moment an answer arrives.
Can you run tcpdump or wireshark to match the timing when the packet appears on the tap-win32 i/f?

After the packet is 'seen' on the 'tap-win32' i/f by the rest of the system there is very little OpenVPN can do - if the rest of the OS takes 500 ms to respond then the rest of the OS (virus scan services? firewall policies? other (rogue) services?) is responsible.

[huebeni]

Hi,

ok here the Traces:

The unencrypted log on the internal interface (TAP):

Code: Select all

No.     Time            Source                Destination           Protocol Info
      1 12:33:52.544263 10.2.124.250          10.2.3.123            ICMP     Echo (ping) request  (id=0x0001, seq(be/le)=134/34304, ttl=128)
      2 12:33:53.064270 10.2.3.123            10.2.124.250          ICMP     Echo (ping) reply    (id=0x0001, seq(be/le)=134/34304, ttl=63)
      3 12:33:53.544329 10.2.124.250          10.2.3.123            ICMP     Echo (ping) request  (id=0x0001, seq(be/le)=135/34560, ttl=128)
      4 12:33:54.064320 10.2.3.123            10.2.124.250          ICMP     Echo (ping) reply    (id=0x0001, seq(be/le)=135/34560, ttl=63)
      5 12:33:54.545380 10.2.124.250          10.2.3.123            ICMP     Echo (ping) request  (id=0x0001, seq(be/le)=136/34816, ttl=128)
      6 12:33:55.056353 10.2.3.123            10.2.124.250          ICMP     Echo (ping) reply    (id=0x0001, seq(be/le)=136/34816, ttl=63)
      7 12:33:55.547377 10.2.124.250          10.2.3.123            ICMP     Echo (ping) request  (id=0x0001, seq(be/le)=137/35072, ttl=128)
      8 12:33:56.059342 10.2.3.123            10.2.124.250          ICMP     Echo (ping) reply    (id=0x0001, seq(be/le)=137/35072, ttl=63)

The encrypted packets on the external if:

Code: Select all

No.     Time            Source                Destination           Protocol Info
     11 12:33:52.544566 192.168.99.174        xx.yyy.zz.134         SSL      Continuation Data
     12 12:33:52.556296 xx.yyy.zz.134         192.168.99.174        SSL      Continuation Data
     13 12:33:52.764225 192.168.99.174        xx.yyy.zz.134         TCP      49509 > https [ACK] Seq=120 Ack=120 Win=64139 Len=0
     16 12:33:53.544653 192.168.99.174        xx.yyy.zz.134         SSL      Continuation Data
     17 12:33:53.556064 xx.yyy.zz.134         192.168.99.174        SSL      Continuation Data
     18 12:33:53.764244 192.168.99.174        xx.yyy.zz.134         TCP      49509 > https [ACK] Seq=239 Ack=239 Win=64020 Len=0
     19 12:33:54.545646 192.168.99.174        xx.yyy.zz.134         SSL      Continuation Data
     20 12:33:54.557070 xx.yyy.zz.134         192.168.99.174        SSL      Continuation Data
     21 12:33:54.756307 192.168.99.174        xx.yyy.zz.134         TCP      49509 > https [ACK] Seq=358 Ack=358 Win=63901 Len=0
     25 12:33:55.547677 192.168.99.174        xx.yyy.zz.134         SSL      Continuation Data
     26 12:33:55.559369 xx.yyy.zz.134         192.168.99.174        SSL      Continuation Data
     27 12:33:55.759287 192.168.99.174        xx.yyy.zz.134         TCP      49509 > https [ACK] Seq=477 Ack=477 Win=63782 Len=0

For me the delay seems to be before the TCPv4_CLIENT READ [117] from 1.2.3.134:443: P_DATA_V1 kid=0 DATA len=116

I decativated all virus scanners, firewalls and other services connected to the interfaces and saw no changes.

I just have no idea.

Michael

[dazo]

My immediate hunch is that it's caused by your TCP setup. TCP uses something often referred to as the Nagle algorithm [1], which collects a certain number of TCP packets before sending them over the wire. And ICMP (ping) packets are small packets, which could easily be hit by this.

Try adding --socket-flags TCP_NODELAY to your config files. That should turn off the Nagle algorithm. From the man page:

Code: Select all

       --socket-flags flags...
              Apply the given flags to the OpenVPN transport socket.  Currently,  only  TCP_NODELAY
              is supported.

              The TCP_NODELAY socket flag is useful in TCP mode, and causes the kernel to send tun‐
              nel packets immediately over the TCP  connection  without  trying  to  group  several
              smaller  packets into a larger packet.  This can result in a considerably improvement
              in latency.

              This option is pushable from server to client, and should be used on both client  and
              server for maximum effect.

[1] http://en.wikipedia.org/wiki/Nagel_algorithm

[huebeni]

Hi there,

finally I solved the BIG problem.
Yes, it is not an OpenVPN problem or Windows 7 problem. It was the == ESET Smart Security 4 Firewall ==.

Source: http://www.wilderssecurity.com/showpost.php?p=1360776

Seems like i found the setting that made it work for me without the slow response when using https/ssl. When you enter the "Advanced Setup" then go to "Protocol Filtering -> SSL" and enable SSL protocol scanning. Then go to "Web access protection -> HTTP, HTTPS" and check "Do not use HTTPs protocol checking". When this setting is checked you have to go back to "Protocol filtering -> SSL" and disable "SSL protocol scanning" again. At least this worked for me on two separate computers.

I assume this is a bug because when SSL protocol scanning is disabled the HTTPs filtering mode settings shouldn't be relevant and the checkbox is disabled as well so unable to change it unless SSL protocol scanning is enabled.

The excluded certificates settings seems to have no effect at all so i assume this one isn't working yet.

Yes, an greyed-out option does the trick. The firewall intercepted the incoming packets (even if switched off) and delayed them.

Now I have ping around 11-12ms.

Thanks for your help. Hope this will help someone else, too.

Michael

[janjust]

Ouch! I was afraid it would be something like that.
Thanks for posting the solution, it will remain visible on this forum for others to see.