Hugely asymmetric TCP speed in tunnel with OpenVPN 2.6.6

[kobuki]

I've been struggling with a weird problem: speeds from the client to the server are in the 600-700 Mbps range, but the opposite direction (server to client) is very slow, about 20-30 Mbps, using a single TCP stream through iperf3 within the tunnel. DCO is active on the server side, but not on the client side. The server is a KVM VM on Proxmox, Debian 12, OpenVPN 2.6.6. The client is an AWS EC2 instance (t3a.medium), also KVM, Debian 12, OpenVPN 2.6.6. UDP speeds are fine, in the 2-300 Mbps range in both directions through the tunnel. RTT is around 20 ms between the peers. Direct connections without VPN are fine, 600-700 Mbps in both directions, sometimes more.

Whatever optimization I try is useless for the "slow direction" (mssfix, sndbuf, rcvbuf, txqueuelen, tun-mtu). They are effective for the "fast direction", but that's not important now. I'm posting the configs and some metrics. Any ideas are welcome. I noticed that the TCP window is a lot smaller in the slower direction. This is probably normal, but I have a hunch that the latency induces a bad windows scaling behavior on the server.

The side running the OpenVPN server:

Code: Select all

# iperf3 -p 51111 -c 172.16.92.101 -R
Connecting to host 172.16.92.101, port 51111
Reverse mode, remote host 172.16.92.101 is sending
[  5] local 172.16.92.1 port 46120 connected to 172.16.92.101 port 51111
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  35.2 MBytes   295 Mbits/sec
[  5]   1.00-2.00   sec  79.7 MBytes   668 Mbits/sec
[  5]   2.00-3.00   sec  76.5 MBytes   642 Mbits/sec
[  5]   3.00-4.00   sec  82.2 MBytes   689 Mbits/sec
[  5]   4.00-5.00   sec  83.9 MBytes   705 Mbits/sec
[  5]   5.00-6.00   sec  83.3 MBytes   699 Mbits/sec
[  5]   6.00-7.00   sec  84.1 MBytes   706 Mbits/sec
[  5]   7.00-8.00   sec  83.5 MBytes   700 Mbits/sec
[  5]   8.00-9.00   sec  83.7 MBytes   702 Mbits/sec
[  5]   9.00-10.00  sec  85.2 MBytes   715 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   781 MBytes   653 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   777 MBytes   652 Mbits/sec                  receiver

iperf Done.
# iperf3 -p 51111 -c 172.16.92.101
Connecting to host 172.16.92.101, port 51111
[  5] local 172.16.92.1 port 37500 connected to 172.16.92.101 port 51111
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.80 MBytes  23.5 Mbits/sec  100   63.2 KBytes
[  5]   1.00-2.00   sec  3.39 MBytes  28.5 Mbits/sec    0   93.5 KBytes
[  5]   2.00-3.00   sec  3.70 MBytes  31.1 Mbits/sec   18   60.6 KBytes
[  5]   3.00-4.00   sec  3.27 MBytes  27.4 Mbits/sec    0   90.8 KBytes
[  5]   4.00-5.00   sec  2.96 MBytes  24.8 Mbits/sec    9   65.8 KBytes
[  5]   5.00-6.00   sec  2.53 MBytes  21.2 Mbits/sec    1   68.5 KBytes
[  5]   6.00-7.00   sec  2.65 MBytes  22.3 Mbits/sec    5   75.0 KBytes
[  5]   7.00-8.00   sec  3.21 MBytes  26.9 Mbits/sec    4   79.0 KBytes
[  5]   8.00-9.00   sec  3.02 MBytes  25.4 Mbits/sec    6   81.6 KBytes
[  5]   9.00-10.00  sec  2.90 MBytes  24.3 Mbits/sec    2   57.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  30.4 MBytes  25.5 Mbits/sec  145             sender
[  5]   0.00-10.02  sec  30.0 MBytes  25.1 Mbits/sec                  receiver

iperf Done.

The side running the OpenVPN client:

Code: Select all

-----------------------------------------------------------
Server listening on 51111 (test #2)
-----------------------------------------------------------
Accepted connection from 172.16.92.1, port 46104
[  5] local 172.16.92.101 port 51111 connected to 172.16.92.1 port 46120
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  36.9 MBytes   309 Mbits/sec    0   1.98 MBytes
[  5]   1.00-2.00   sec  80.0 MBytes   671 Mbits/sec    0   3.15 MBytes
[  5]   2.00-3.00   sec  76.2 MBytes   640 Mbits/sec    0   3.15 MBytes
[  5]   3.00-4.00   sec  82.5 MBytes   692 Mbits/sec    0   3.15 MBytes
[  5]   4.00-5.00   sec  83.8 MBytes   703 Mbits/sec    0   3.15 MBytes
[  5]   5.00-6.00   sec  82.5 MBytes   692 Mbits/sec    0   3.15 MBytes
[  5]   6.00-7.00   sec  85.0 MBytes   713 Mbits/sec    0   3.15 MBytes
[  5]   7.00-8.00   sec  82.5 MBytes   692 Mbits/sec    0   3.15 MBytes
[  5]   8.00-9.00   sec  83.8 MBytes   703 Mbits/sec    0   3.15 MBytes
[  5]   9.00-10.00  sec  85.0 MBytes   713 Mbits/sec    0   3.15 MBytes
[  5]  10.00-10.02  sec  2.50 MBytes  1.01 Gbits/sec    0   3.15 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   781 MBytes   653 Mbits/sec    0             sender
-----------------------------------------------------------
Server listening on 51111 (test #3)
-----------------------------------------------------------
Accepted connection from 172.16.92.1, port 37484
[  5] local 172.16.92.101 port 51111 connected to 172.16.92.1 port 37500
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.19 MBytes  18.4 Mbits/sec
[  5]   1.00-2.00   sec  3.41 MBytes  28.6 Mbits/sec
[  5]   2.00-3.00   sec  3.75 MBytes  31.5 Mbits/sec
[  5]   3.00-4.00   sec  3.22 MBytes  27.0 Mbits/sec
[  5]   4.00-5.00   sec  2.92 MBytes  24.5 Mbits/sec
[  5]   5.00-6.00   sec  2.74 MBytes  23.0 Mbits/sec
[  5]   6.00-7.00   sec  2.67 MBytes  22.4 Mbits/sec
[  5]   7.00-8.00   sec  3.06 MBytes  25.7 Mbits/sec
[  5]   8.00-9.00   sec  3.13 MBytes  26.2 Mbits/sec
[  5]   9.00-10.00  sec  2.90 MBytes  24.3 Mbits/sec
[  5]  10.00-10.02  sec  57.9 KBytes  22.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec  30.0 MBytes  25.1 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 51111 (test #4)
-----------------------------------------------------------

# ping gw.server.tld  # running from the client
PING gw.server.tld (1.2.106.211) 56(84) bytes of data.
64 bytes from 1.2.106.211 (1.2.106.211): icmp_seq=1 ttl=53 time=20.4 ms
64 bytes from 1.2.106.211 (1.2.106.211): icmp_seq=2 ttl=53 time=20.1 ms
64 bytes from 1.2.106.211 (1.2.106.211): icmp_seq=3 ttl=53 time=20.4 ms
64 bytes from 1.2.106.211 (1.2.106.211): icmp_seq=4 ttl=53 time=20.2 ms
64 bytes from 1.2.106.211 (1.2.106.211): icmp_seq=5 ttl=53 time=20.3 ms
^C
--- gw.server.tld ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 20.115/20.280/20.409/0.120 ms

Server config:

View OriginalServer config

proto udp
port 11940
tls-server
dev tun1
keepalive 10 120
persist-key
persist-tun
verb 3

dh none
tls-groups secp521r1
auth SHA512
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
cipher AES-256-GCM
data-ciphers AES-256-GCM
tls-version-min 1.2

ifconfig-pool-persist /var/lib/openvpn/ipp.txt
status /var/lib/openvpn/status.log
log-append /var/log/openvpn/openvpn.log

topology subnet
server 172.16.93.0 255.255.255.0
push "dhcp-option DNS 10.92.40.254"

sndbuf 0
rcvbuf 0
txqueuelen 4000
tun-mtu 1400

<ca>
</ca>

<cert>
</cert>

<key>
</key>

<tls-crypt>
</tls-crypt>

Client config:

View OriginalClient config

client
dev tun
remote gw.server.tld 11940 udp
resolv-retry infinite
nobind
remote-cert-tls server
auth SHA512
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
cipher AES-256-GCM
tls-version-min 1.2
persist-key
persist-tun
verb 3
auth-nocache
explicit-exit-notify

sndbuf 0
rcvbuf 0
txqueuelen 4000
tun-mtu 1400

<ca>
</ca>

<cert>
</cert>

<key>
</key>

<tls-crypt>
</tls-crypt>

[kobuki]

Well, some more info. I fired up a simple WireGuard instance between the peers. It provides about 400-500 Mbits/s bandwdth in both directions. Replacing OVPN with another VPN solution is not an option, I just wanted to show that the system is capable of pretty good speeds in either direction. I'm using OpenVPN at many customers and I always advocate its use because of its strengths, but to shade things a bit, I almost always need to employ a few tunings and system-specific settings to get it perform adequately. It's getting a bit tiring, to be honest.

I'd be definitely glad to hear some opinions.

[kobuki]

I think it has been fixed, or so it seems. The key was to use the dco-v2 module on both ends. I think it's still at least a regression, since without DCO on one end shouldn't trample the speed too so much, down to a few dozen Mbits from several hundred in the other direction. Now I consistently get a BW between 400 and 500 Mbits/s or more with a single TCP stream only, in both directions.