Slow outbound network speed for Windows Server 2016 only via the OpenVPN tunnel

rmarilyak

Hello,

I've faced a very strange issue with slow outbound network speed from Windows Server 2016 Standard server via the OpenVPN tunnel.
OpenVPN server is Windows Server 2012 R2, client is Windows Server 2016. The inbound network speed for Windows Server 2016 is great.
But the outbound network speed is nearly 30-40 kbps. I've got the same results using several tests: iperf testings, file download via SMB, Web-based downloading (using HTTP) etc.

You can find server and client configs below:

mode server
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
auth SHA256
tls-auth <path-to-key-file>\\tls-auth.key 0
remote-cert-eku "TLS Web Client Authentication"
local <local-ip-address>
port 1197
proto udp
dev tun
dev-node <tap-adapter-name>
ca <path-to-ca-cert-file>\\ca.crt
cert <path-to-server-cert-file>\\server.crt
key <path-to-server-key-file>\\server.key
dh <path-to-dh-params-file>\\dh4096.pem
ifconfig 172.16.144.1 172.16.144.2
ifconfig-pool 172.16.144.4 172.16.144.20
ifconfig-pool-persist <path-to-persistent-routes-file>\\persistent-routes.txt
route 10.0.44.0 255.255.255.0
route 172.16.144.0 255.255.255.0
push "route 172.16.144.1"
push "route 10.0.4.0 255.255.255.0"
push "route 172.16.144.0 255.255.255.0"
client-config-dir C:\\OpenVPN\\ccd
keepalive 10 60
cipher AES-256-CBC
comp-lzo
max-clients 1
persist-key
persist-tun
status C:\\OpenVPN\\log\\server-status.log
log C:\\OpenVPN\\log\\server.log
verb 3

tls-client
tls-version-min 1.2
cipher AES-256-CBC
tls-auth <path-to-tls-auth-file>\\tls-auth.key 1
verify-x509-name 'C=<Country>, ST=<State>, L=<City>, O=<Organization>, OU=<OrganizationalUnit>, CN=<CommonName>, name=<Name>, emailAddress=<Email>' subject
remote-cert-eku "TLS Web Server Authentication"
auth SHA256
dev tun
dev-node <tap-adapter-name>
proto udp
remote <remote-server-ip-address> 1197
pull
resolv-retry infinite
nobind
persist-key
persist-tun
ca <path-to-server-ca-file>\\ca.crt
cert <path-to-client-cert-file>\\client.crt
key <path-to-client-key-file>\\client.key
remote-cert-tls server
comp-lzo
log C:\\OpenVPN\\log\\client.log
verb 3

The tunnels is getting up and it works greatly, but only in one direction - from Windows Server 2012 R2 to Windows Server 2016.
I've been using such server-client configurations setup for several years with Windows Server 2012 R2 servers and I've never faced such issue before.
At first I thought that our ISP has some network limitations, but it turned out that the same tests shows great network speed results using the public IP addresses in both directions.
The issue only occurs inside the VPN tunnel. I've spent 3 days tryng to figure it out, but failed. I've installed all latest Windows updates, reinstalled OpenVPN, tried to switch from UDP to TCP,
played with performance settings in configs (link-mtu, sndbuf, rcvbuf etc) but still no luck. I've tested the same setup between two Windows Server 2012 R2 servers and it works greatly in both directions.
Then I've tested it with another Windows Server 2016 Standard server (different server and different ISP) and it showed the same awful results in outbound direction.
When I've set the same OpenVPN tunnel between two Windows Server 2016 Standard servers I've got the same poor network speed in both directions.

After all my efforts I started network debugging with Wireshark and I've noticed a huge amount of such errors:

[TCP Previous segment not captured]
[TCP Dup ACK]
[TCP Retransmission]

I believe that the issue is somehow related only to the Windows Server 2016 version and I am more than confident that it depends on server's TCP stack settings.
I've noticed that Windows Server 2016 has a congestion control provider setting set to "default", while previous versions of Windows has this setting set to "none".

C:\Users\Administrator> netsh int tcp show global
......
Add-On Congestion Control Provider : default
......

Also, I have one Windows Server 2016 server that somehow has this congestion control provider setting set to "none", and such issue does not reproduce on it!
But I cannot change this setting to "none" on other Windows Server 2016 hosts, it simply does not have a possibility to choose such value for it.

Does anyone have any clue how can I resolve the issue? What did I missed?
I would be very grateful for any help.

TinCanTech

Have you checked your log files ?

rmarilyak

Dear TinCanTech,

What logs are you talking about? VPN logs or Windows Events Logs?
I've definitely checked both of them from the very beginning and found nothing useful to solve the issue.
As I've mentioned, the VPN tunnel between the servers successfully starts up and I do not have any issues in inbound direction.
It is also working in outbound direction too, but the speed is about 30-40 kbps. The issue occurs only if one server is on Windows Server 2016.
The same configs works great won Windows Server 2012 R2 servers

TinCanTech

rmarilyak wrote What logs are you talking about?

Your openvpn logs ..

Anyway ..

rmarilyak wrote The issue occurs only if one server is on Windows Server 2016.
The same configs works great won Windows Server 2012 R2 servers

rmarilyak wrote I believe that the issue is somehow related only to the Windows Server 2016 version and I am more than confident that it depends on server's TCP stack settings.
I've noticed that Windows Server 2016 has a congestion control provider setting set to "default", while previous versions of Windows has this setting set to "none".
C:\Users\Administrator> netsh int tcp show global
......
Add-On Congestion Control Provider : default
......

At least you may have found the cause of the problem but I don't know how this relates to or effects openvpn. I have asked some of the devs to take a look at this thread and I'll let you know if they have anything useful to add.

rmarilyak

Dear TinCanTech,

Thank you so much. Looking forward for any update from you.

TinCanTech

It was suggested that you post to the developer mailing list and reference this thread.

rmarilyak

Dear TinCanTech,

Thank you for the update.
I've performed some additional debug and found lots of such lines in OpenVPN logs:

Wed Oct 03 15:37:09 2018 us=831726 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:37:10 2018 us=909906 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:37:10 2018 us=909906 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:37:11 2018 us=878707 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:38:55 2018 us=352851 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:39:02 2018 us=472169 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped
Wed Oct 03 15:39:03 2018 us=550344 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped

Could this be the cause of my issue? I am not 100% sure about that.
I do have a ccd settings for internal network, but this lines shows the client's public IP address and I don't know how can I fix this.

rmarilyak

Dear TinCanTech,

Can you please explain more detailed how can I post to the developer mailing list? I've followed your link and went to https://sourceforge.net/p/openvpn/mailman/openvpn-devel/
but I do not see how can I post there.
Please advise.

TinCanTech

rmarilyak wrote Wed Oct 03 15:37:09 2018 us=831726 <client-name>/62.212.85.XX:54163 MULTI: bad source address from client [62.212.85.XX], packet dropped

This can be safely ignored ..

They are stray packets from the wrong source address and openvpn simply drops them

rmarilyak wrote Can you please explain more detailed how can I post to the developer mailing list?

You must subscribe and send email.

rmarilyak

Dear TinCanTech,

Got it. Thank you for help