tcp retransmission on windows client via openvpn tunnel

[skywalker]

Hi guys ,I have a problem with TCP connection via OpenVPN tunnel .the OpenVPN service is running on CentOS8.2 and the Client software is OpenVPN connect 3.2.2 running on windows 10.
here is the topology:

LAN A--(Client A CentOS8.2) ------------OpenVPN Server(CentOS8.2) ----------Client B (Linux )
########################################|
########################################|------------------------------Client C(Windows 10)
I do everything ok on the client B like accessing the webserver on LAN A ,or SSH login to OpenVPN Server via its LAN address. but I just can't do it on Client C. On Client C ping is all ok and even the TCP three handshake is ok but can't login to the ssh service or access the website. and wireshark showed that after three handshake between client C and Clint A or the OpenVPN server,something happened and caused TCP Retransmission and TCP dup ACK and finally the Server side send RST packet to finish the connection.

View OriginalServer config

port 1194
proto udp
dev tun
ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/automation-lab.crt
key /etc/openvpn/server/automation-lab.key
dh /etc/openvpn/server/dh.pem
server 172.16.100.0 255.255.255.0
push "172.19.248.0 255.255.240.0"
push "172.19.167.0 255.255.240.0"
push "dhcp-option DNS 114.114.114.114"
push "dhcp-option DNS 114.114.114.114"
client-config-dir /etc/openvpn/ccd
route 192.168.15.0 255.255.255.0
route 192.168.16.0 255.255.255.0
duplicate-cn
cipher AES-256-CBC
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-128-CBC-SHA256
auth SHA512
auth-nocache
keepalive 20 60
persist-key
persist-tun
compress lz4
daemon
user nobody
group nobody
log-append /var/log/openvpn.log
verb 3
client-to-client
#tun-mtu 1500
mssfix 1550

View OriginalClient config

client
dev tun
proto udp
remote xxx.xxx.xxx.xxx 1194
ca ca.crt
cert client.crt
key client.key
cipher AES-256-CBC
auth SHA512
auth-nocache
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-128-CBC-SHA256
resolv-retry infinite
compress lz4
nobind
persist-key
persist-tun
mute-replay-warnings
verb 7
route 172.19.248.132 255.255.255.255 vpn_gateway
route 192.168.16.0 255.255.255.0 vpn_gateway
log d:\\client.log
mss_fix 1550

tcpdump result :

Code: Select all

15:49:54.987889 IP 172.16.100.6.11667 > 172.19.248.132.ssh: Flags [S], seq 3553131263, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:49:54.987949 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [S.], seq 2104630248, ack 3553131264, win 29200, options [mss 1460,nop,nop,sackOK], length 0
15:49:55.031834 IP 172.16.100.6.11667 > 172.19.248.132.ssh: Flags [.], ack 1, win 64240, length 0
15:49:55.042991 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 1:22, ack 1, win 29200, length 21
15:49:55.085022 IP 172.16.100.6.11667 > 172.19.248.132.ssh: Flags [P.], seq 1:51, ack 22, win 64219, length 50
15:49:55.085078 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [.], ack 51, win 29200, length 0
15:49:55.087080 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 51, win 29200, length 1048
15:49:55.097878 IP 172.16.100.6.11667 > 172.19.248.132.ssh: Flags [P.], seq 1511:1859, ack 22, win 64219, length 348
15:49:55.097915 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [.], ack 51, win 30016, options [nop,nop,sack 1 {1511:1859}], length 0
15:49:55.098608 IP 172.16.100.6.11667 > 172.19.248.132.ssh: Flags [.], seq 51:1511, ack 22, win 64219, length 1460
15:49:55.098634 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [.], ack 1859, win 32120, length 0
15:49:55.332844 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:49:55.580857 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:49:56.076844 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:49:57.076862 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:49:59.060853 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:50:03.028855 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:50:11.348854 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:50:27.220854 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048
15:50:58.964855 IP 172.19.248.132.ssh > 172.16.100.6.11667: Flags [P.], seq 22:1070, ack 1859, win 32120, length 1048

and I also change the MTU on window 10 ,to let both side have the same MSS.

Code: Select all

C:\WINDOWS\system32>netsh interface ipv4 show subinterfaces

   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
4294967295                1          0     477776  Loopback Pseudo-Interface 1
  1500                1      44091     430304  loca lconnection
  1500                1  66785230436  28797575366  WLAN
  1500                5          0          0  ethernet
  1500                5          0          0  TUN adapter

TCP Global Parameters

Code: Select all

C:\WINDOWS\system32>netsh int tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : default
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled
Initial RTO                         : 1000
Receive Segment Coalescing State    : enabled
Non Sack Rtt Resiliency             : disabled
Max SYN Retransmissions             : 4
Fast Open                           : enabled
Fast Open Fallback                  : enabled
HyStart                             : enabled
Pacing Profile                      : off

I also changed the sysctl parameter on Linux server:

Code: Select all

add below to sysctl.conf
net.ipv4.tcp_window_scaling=0
and run 
sysctl -p

SSH communication :

Code: Select all

No.	Time	Source	Destination	Protocol	Length	Info
4	0.054723	172.19.248.132	172.16.100.6	SSHv2	61	Server: Protocol (SSH-2.0-OpenSSH_8.0)
5	0.096723	172.16.100.6	172.19.248.132	SSHv2	90	Client: Protocol (SSH-2.0-nsssh2_6.0.0038 NetSarang Computer, Inc.)
7	0.098509	172.19.248.132	172.16.100.6	SSHv2	1088	Server: Key Exchange Init
9	0.126056	172.16.100.6	172.19.248.132	SSHv2	388	Client: Key Exchange Init
1	0.000000	172.16.100.6	172.19.248.132	TCP	52	10050 → 22 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
2	0.000057	172.19.248.132	172.16.100.6	TCP	48	22 → 10050 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1
3	0.042025	172.16.100.6	172.19.248.132	TCP	40	10050 → 22 [ACK] Seq=1 Ack=1 Win=64240 Len=0
6	0.096764	172.19.248.132	172.16.100.6	TCP	40	22 → 10050 [ACK] Seq=22 Ack=51 Win=29200 Len=0
8	0.125998	172.16.100.6	172.19.248.132	TCP	1500	10050 → 22 [ACK] Seq=51 Ack=22 Win=64219 Len=1460 [TCP segment of a reassembled PDU]
10	0.126075	172.19.248.132	172.16.100.6	TCP	40	22 → 10050 [ACK] Seq=1070 Ack=1859 Win=35040 Len=0
11	0.344979	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
12	0.593000	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
13	1.080979	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
14	2.056985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
15	4.040985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
16	7.944985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
17	15.944999	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
18	31.816984	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048

http access before I change the MSS value

Code: Select all

No.	Time	Source	Destination	Protocol	Length	Info
1	0.000000	172.16.100.6	192.168.16.2	TCP	66	50937 → 20080 [SYN] Seq=0 Win=65535 Len=0 MSS=1357 WS=256 SACK_PERM=1
2	0.000135	192.168.16.2	172.16.100.6	TCP	66	20080 → 50937 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
3	0.095213	172.16.100.6	192.168.16.2	TCP	60	50937 → 20080 [ACK] Seq=1 Ack=1 Win=1573888 Len=0
4	0.103142	172.16.100.6	192.168.16.2	HTTP	490	GET / HTTP/1.1 
5	0.103254	192.168.16.2	172.16.100.6	TCP	54	20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=0
6	0.105846	192.168.16.2	172.16.100.6	TCP	2768	20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=2714 [TCP segment of a reassembled PDU]
7	0.105860	192.168.16.2	172.16.100.6	HTTP	2305	HTTP/1.1 200 OK  (text/html)
8	0.304352	192.168.16.2	172.16.100.6	TCP	948	[TCP Retransmission] 20080 → 50937 [PSH, ACK] Seq=4072 Ack=437 Win=30336 Len=894
9	0.608388	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
10	1.240393	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
11	2.456362	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
12	4.888380	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
13	9.688521	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
14	15.188923	172.16.100.6	192.168.16.2	TCP	60	[TCP Keep-Alive] 50937 → 20080 [ACK] Seq=436 Ack=1 Win=1573888 Len=1
15	15.189028	192.168.16.2	172.16.100.6	TCP	66	[TCP Keep-Alive ACK] 20080 → 50937 [ACK] Seq=4966 Ack=437 Win=30336 Len=0 SLE=436 SRE=437
16	19.416346	192.168.16.2	172.16.100.6	TCP	1411	[TCP Retransmission] 20080 → 50937 [ACK] Seq=1 Ack=437 Win=30336 Len=1357
17	30.106827	192.168.16.2	172.16.100.6	TCP	54	20080 → 50937 [FIN, ACK] Seq=4966 Ack=437 Win=30336 Len=0
18	30.191704	172.16.100.6	192.168.16.2	TCP	60	[TCP Dup ACK 3#1] 50937 → 20080 [ACK] Seq=437 Ack=1 Win=1573888 Len=0

and it just didn't work.

Any ideas on what might be wrong please?

Thank you

[300000]

turn off windows firewall and try it again. your server have push "172.19.248.0 255.255.240.0" so how client you add" route 172.19.248.132 255.255.255.255 vpn_gateway" there are something in here not right . the whole route 172.19.248.0 have push down to client so every address in that range can access from client

[skywalker]

turn off windows firewall and try it again. your server have push "172.19.248.0 255.255.240.0" so how client you add" route 172.19.248.132 255.255.255.255 vpn_gateway" there are something in here not right . the whole route 172.19.248.0 have push down to client so every address in that range can access from client

thank you for reply.
the windows firewall is off .and I think it's not a route problem because the ping to all these addresses is ok .I can telnet port 22 of 172.19.248.132 and port 8080 of 192.168.16.2 on the windows 10 client. and even the tcp three handshake can establish successfully but it seems something went wrong after that ,the server just keep sending retransmission packets to the client and sometimes showed " tcp dup ack ".I think something caused the client couldn't send back some ack packets to the server but I don't know why.

[TinCanTech]

########################################|------------------------------Client C(Windows 10)
I do everything ok on the client B like accessing the webserver on LAN A ,or SSH login to OpenVPN Server via its LAN address. but I just can't do it on Client C

What do you use on Windows for ssh ?

[skywalker]

########################################|------------------------------Client C(Windows 10)
I do everything ok on the client B like accessing the webserver on LAN A ,or SSH login to OpenVPN Server via its LAN address. but I just can't do it on Client C

What do you use on Windows for ssh ?

I mean I ssh to the LAN address of OpenVPN server through OpenVPN tunnel on client side which is windows 10. usually I use putty or xshell to ssh to the server.
I captured these packets and on the wireshark it shows like below:

Code: Select all

No.	Time	Source	Destination	Protocol	Length	Info
4	0.054723	172.19.248.132	172.16.100.6	SSHv2	61	Server: Protocol (SSH-2.0-OpenSSH_8.0)
5	0.096723	172.16.100.6	172.19.248.132	SSHv2	90	Client: Protocol (SSH-2.0-nsssh2_6.0.0038 NetSarang Computer, Inc.)
7	0.098509	172.19.248.132	172.16.100.6	SSHv2	1088	Server: Key Exchange Init
9	0.126056	172.16.100.6	172.19.248.132	SSHv2	388	Client: Key Exchange Init
1	0.000000	172.16.100.6	172.19.248.132	TCP	52	10050 → 22 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
2	0.000057	172.19.248.132	172.16.100.6	TCP	48	22 → 10050 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1
3	0.042025	172.16.100.6	172.19.248.132	TCP	40	10050 → 22 [ACK] Seq=1 Ack=1 Win=64240 Len=0
6	0.096764	172.19.248.132	172.16.100.6	TCP	40	22 → 10050 [ACK] Seq=22 Ack=51 Win=29200 Len=0
8	0.125998	172.16.100.6	172.19.248.132	TCP	1500	10050 → 22 [ACK] Seq=51 Ack=22 Win=64219 Len=1460 [TCP segment of a reassembled PDU]
10	0.126075	172.19.248.132	172.16.100.6	TCP	40	22 → 10050 [ACK] Seq=1070 Ack=1859 Win=35040 Len=0
11	0.344979	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
12	0.593000	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
13	1.080979	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
14	2.056985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
15	4.040985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
16	7.944985	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
17	15.944999	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048
18	31.816984	172.19.248.132	172.16.100.6	TCP	1088	[TCP Retransmission] 22 → 10050 [PSH, ACK] Seq=22 Ack=1859 Win=35040 Len=1048

[TinCanTech]

mssfix 1550

Can you remember why you used this ?

[skywalker]

mssfix 1550

Can you remember why you used this ?

first I thought it was because the mss parameter on the client side is smaller than the server side and triggered tcp fragment .and it turns out this doesn't work and in the end I use "netsh interface ipv4 set subinterface "tun_adapter" mtu=1500 store=persistent" to change the client side MTU and and it works . after this the MSS parameter on both sides are 1460 ,and it still has the tcp retransmission issue .

[TinCanTech]

Your mssfix setting clearly indicates that you do not understand that side of networking. The simplest and most logical answer for you is to not use those advanced network options. Openvpn has perfectly suitable defaults for most networks.

[skywalker]

Your mssfix setting clearly indicates that you do not understand that side of networking. The simplest and most logical answer for you is to not use those advanced network options. Openvpn has perfectly suitable defaults for most networks.

I didn't use these settings at first .I used the default settings before and it didn't work.so I tried to change some parameters .

[skywalker]

This problem has been solved now.

[TinCanTech]

Default settings .....