Slow Speed From Cloud Image

RyanP

Helo all,
I am setting up a POC using the Oracle cloud image https://openvpn.net/oracle-cloud/ for our dev environment. The cloud instance is capable of 500Mbps and my test client can do 1000Mbps, but with the default config while trying to copy a 1GB file over scp from client to dev server I am only able to sustain 20Mbps throughput . Access server version is 2.8.3. I have tested the client config with Windows, Mac and Linux. Any help would be much appreciated. Thank you.

chilinux

These official cloud images do not get updated very often?

AS v2.8.3 is from March 2020. Between AS and the bundled clients packages, there has been at least 5 security fixes since then.

It would be nice if OpenVPN AS would update the cloud images at least twice a year. Or even better add a first-run auto-update.

Not really related to your throughput issue, but consider upgrading to at least AS v2.8.8 (latest is v2.9.2).

SCP over OpenVPN is not the best way to troubleshoot throughput. Here is the process the 1GB file is going through:

SSH/SCP client encrypts each packet of the file
OpenVPN client re-encrypts the SSH packet
OpenVPN server decrypts the the VPN packet before transmitting the SSH packet
Dev server decrypts the SSH packet

The total end result is 20 Mbps. At what point is the throughput being capped? I have no clue.

Can you provide details of the throughput of the same SCP perform without OpenVPN?

Can you run rsync in --daemon mode on the Dev server and provide what the throughput is from client to dev over OpenVPN without SSH?

Can you do the same rsync test without both OpenVPN and SSH and provide the throughput?

Can you run the following on the client, vpn server and dev server:

openssl speed -seconds 1 aes

If you can provide more information, that will help in figuring out where to focus efforts to further troubleshoot.

RyanP

So I updated the AS server to 2.9.2, and the issue persists.

scp speed w/o AS server are ~300Mbps with 1GB test file

scp speed tunneling through AS server to dev server is ~192Mbps

iperf3 between dev server and AS server:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.45 GBytes 1.25 Gbits/sec 249 sender
[ 4] 0.00-10.00 sec 1.45 GBytes 1.25 Gbits/sec receiver

iperf3 between linux client and dev server over VPN:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 18.4 MBytes 15.4 Mbits/sec 173 sender
[ 5] 0.00-10.00 sec 17.8 MBytes 15.0 Mbits/sec receiver

openssl speed -seconds 1 aes AS server:
OpenSSL 1.1.1 11 Sep 2018
built on: Mon Mar 22 11:42:42 2021 UTC
options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-nwsL4a/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128 cbc 109086.38k 118987.39k 121550.85k 268042.24k 270442.50k 270548.99k
aes-192 cbc 91317.34k 99168.83k 101015.55k 226626.56k 228319.23k 228425.73k
aes-256 cbc 79382.10k 85135.74k 86481.66k 196286.46k 197648.38k 197574.66k

openssl speed -seconds 1 aes Linux client:
OpenSSL 1.1.1k FIPS 25 Mar 2021
built on: Fri Mar 26 00:00:00 2021 UTC
options:bn(64,64) md2(char) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config"
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128 cbc 265930.65k 295321.34k 298899.91k 298920.08k 268267.31k 304047.32k
aes-192 cbc 225291.33k 242592.06k 236430.48k 254334.98k 262963.20k 249905.15k
aes-256 cbc 217109.17k 224766.84k 227086.34k 220051.46k 229268.43k 228442.11k

Pippin

Basically it comes down to the hardware, CPU speed, AES-NI support and bandwidth on each side.

To find a ballpark figure what your hardware is capable of, you could generate a key:

openvpn --genkey --secret /tmp/secret

do this test

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

and calculate:

3200 / real seconds = ~ Mbps

chilinux

The client and VPN server are getting similar enough result for 1024 bytes such that large file transfers shouldn't be drastically different.

Are you connecting to OpenVPN via UDP or TCP?

RyanP

openvpnas@as:~$ time /usr/local/openvpn_as/sbin/openvpn-openssl --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
2021-07-27 17:22:51 DEPRECATED OPTION: The option --secret is deprecated.
2021-07-27 17:22:51 Cipher negotiation is disabled since neither P2MP client nor server mode is enabled

real 0m1.835s
user 0m1.826s
sys 0m0.008s

I still have the AS server in default settings, so it looks like it has the ability to connect to both UDP and TCP. On further inspection, it appears I am connected over UDP.

chilinux

Well, the openvpn test comes to 1.7 Gbps. That isn't your weak link.

My best guess at this point is packet fragmentation but MTU discovery performed by TCP for SCP should address that.

You could try using tcpdump to see if the SCP packets leaving the AS server for the dev server are fragmented or not.

Also, maybe try doing OpenVPN over TCP instead of over UDP and see if that makes any difference.

Given you are getting 192 Mbps when proxying through the AS server to the dev server, I get the feeling you should be seeing at least 150 Mbps while using OpenVPN.

OpenVPN AS support may be able to dig deeper into the problem by checking over the logs.

RyanP

Thanks. Does it make a difference that the AS interface (ens3) MTU is set to 9000 ? I'll switch to TCP only and give that a go. If I continue to experience problems I'll open a ticket with support. Thanks again.

chilinux

Yes, jumbo packets makes a difference.

So, back to square 1 in describing your environment (correct me if I get anything wrong):
You have two cloud servers, an AS cloud server and a Dev cloud server
Both of these are on the same virtual network with jumbo packets (MTU 9000) enabled
And you have external clients that have standard 1500 MTU to their ISP

Then just shooting from the hip here, but I am guessing on either the AS or Dev server (or both) you have iptables/firewall rules to block most ICMP packets from each other? If a ICMP Type 3 (Destination Unreachable) Code 4 (Fragmentation Needed and Don't Fragment was Set) packet is attempted by the AS server's network stack, will it actually get sent to the Dev server, received and processed? Or is it just dropped by a firewall rule along the way?

Is AS set to do NAT or to route an additional pool of client IPs onto the virtual network?

Maybe the best way to improve performance is to establish a route on each of the Dev servers declaring a MTU of 1350 for reaching the AS server IP (if doing NAT) or the client IPs range.

That which is your chosen cloud provider discusses this in their documentation here:
https://docs.oracle.com/en-us/iaas/Content/Network/Troubleshoot/connectionhang.htm

I feel dirty now, I am going to take an "unbreakable" shower to relax.