[Solved]High downstream traffic from v. 2.5.7 server vs v. 2.4.8

Aitarus

Hi.
I'm using two OVPN servers on two different Windows 10 machines and both of them are in the same physical network.
One server is version 2.4.8 and the recent one 2.5.7 and both allow to see other computers on physical network. Strange thing is that except difference between TLS authentication (tls-auth vs tls-crypt-v2) both of them are configured the same, but there is huge difference in downstream traffic between the two.
When I'm connected from the same client device (be it PC or cell phone) to Ovpn server v. 2.5.7 downstream traffic is pretty much 2 time higher than when i'm connected to ovpn server v. 2.4.8. Likewise when I'm downloading a chunk of data from database, on v. 2.4.8 it takes 1.7 MiB (about 14 seconds), but when i reconnect to v. 2.5.7 the same operation takes 3.5-4.0 MiB (about 40-45 seconds). On older server using RDP and downloading lots of data from database (by Grafana for instance) using up like about 80-150 MiB downstram after one hour of work. On new server after few minutes of barely starting RDP session it's already 80 MiB or more.

Could it be that this new tls certificate generates this much traffic?
P.S. Compresion by lz4 is set on both servers.

Pippin

Hi,

No, I wouldn't think.
Sounds puzzling.....

You can test a download of a fixed file size of lets say 50 MiB and time it.
Download from outside of the physical network where the servers are located.
Download from both servers directly and from the database machine (all the same 50 MiB file).

It might be helpful to post logs of the tests of servers and client from the moment the client connects, downloads and disconnects:
https://forums.openvpn.net/viewtopic.php?t=22603#p68963

Aitarus

Thanks for your advice Pippin.

I downloaded 50 MiB CSV file from both PCs to client PC outside local network. But before that, I copied config files etc. from 2.4.8 to 2.5.7, so both servers were running on the same configuration.

Config file for server is:

port 3791
proto udp
dev tun
ca [path]\\ca.crt
cert [path]\\server.crt
key [path]\\server.key
dh [path]\\dh4096.pem
topology subnet
server 10.9.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 10.0.0.0 255.255.255.0"
client-to-client
keepalive 10 120
tls-auth [path]\\ta.key 0
cipher AES-256-GCM
compress lz4-v2
push "compress lz4-v2"
persist-key
persist-tun
status openvpn-status.log
verb 3

Client config is:

client
dev tun
proto udp
remote [IP] 3791
resolv-retry infinite
nobind
persist-key
persist-tun
ca [path]\\ca.crt
cert [path]\\client.crt
key [path]\\client.key
remote-cert-tls server
tls-auth [path]\\ta.key" 1
cipher AES-256-GCM
comp-lzo
verb 3

Results of downloading files using Robocopy:

Bytes : 50.18 m
Times : 0:03:27
Speed : 254292 Bytes/sec.
Speed : 14.550 MegaBytes/min.
OVPN Client GUI: received 20.8 MiB

Bytes : 50.18 m
Times : 0:08:16
Speed : 106149 Bytes/sec.
Speed : 6.073 MegaBytes/min.
OVPN Client GUI: received: 53.2 MiB

Could it be that something within Windows causes this (like retransmissions or such)? Or maybe Ovpn won't compress packets before sending
them?
In addition, below is a snippet from Wireshark log during file download. Wireshark was listening on Windows-TAP device on client side.

10.0.0.45 Server PC's address
10.9.0.1 Ovpn server
10.9.0.2 Ovpn connected client

      1 0.000000       10.9.0.2              10.0.0.45             SMB2     234    Create Request File: 
      2 0.057598       10.0.0.45             10.9.0.2              SMB2     298    Create Response File: 
      3 0.057939       10.9.0.2              10.9.0.1              SMB2     275    GetInfo Request FS_INFO/FileFsVolumeInformation File: ;GetInfo Request FS_INFO/FileFsAttributeInformation File: 
      4 0.101856       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=181 Ack=245 Win=1023 Len=0
      5 0.115330       10.9.0.1              10.9.0.2              SMB2     250    GetInfo Response;GetInfo Response
      6 0.115587       10.9.0.2              10.0.0.45             SMB2     146    Close Request File: 
      7 0.164298       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=222 Ack=197 Win=1022 Len=0
      8 0.171474       10.0.0.45             10.9.0.2              SMB2     182    Close Response
      9 0.177765       10.9.0.2              10.9.0.1              SMB2     274    Create Request File: [Path]\Desktop\qqq
     10 0.217794       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=273 Ack=373 Win=1023 Len=0
     11 0.236197       10.9.0.1              10.9.0.2              SMB2     298    Create Response File: [Path]\Desktop\qqq
     12 0.236563       10.9.0.2              10.0.0.45             SMB2     146    Close Request File: [Path]\Desktop\qqq
     13 0.286897       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=442 Ack=441 Win=1021 Len=0
     14 0.292626       10.0.0.45             10.9.0.2              SMB2     182    Close Response
     15 0.293035       10.9.0.2              10.9.0.1              SMB2     350    Create Request File: [Path]\Desktop\qqq
     16 0.333749       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=365 Ack=501 Win=1022 Len=0
     17 0.352764       10.9.0.1              10.9.0.2              SMB2     378    Create Response File: [Path]\Desktop\qqq
     18 0.353115       10.9.0.2              10.0.0.45             SMB2     260    Find Request File: [Path]\Desktop\qqq SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *;Find Request File: [Path]\Desktop\qqq SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *
     19 0.402825       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=738 Ack=765 Win=1019 Len=0
     20 0.414282       10.0.0.45             10.9.0.2              SMB2     554    Find Response;Find Response, Error: STATUS_NO_MORE_FILES[Malformed Packet]
     21 0.415458       10.9.0.2              10.9.0.1              SMB2     382    Create Request File: [Path]\Desktop\qqq
     22 0.465378       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=571 Ack=1001 Win=1020 Len=0
     23 0.473022       10.9.0.1              10.9.0.2              SMB2     322    Create Response File: [Path]\Desktop\qqq
     24 0.473422       10.9.0.2              10.0.0.45             SMB2     146    Close Request File: [Path]\Desktop\qqq
     25 0.518791       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=1066 Ack=1033 Win=1024 Len=0
     26 0.529330       10.0.0.45             10.9.0.2              SMB2     182    Close Response
     27 0.533010       10.9.0.2              10.9.0.1              SMB2     382    Create Request File: [Path]\Desktop\qqq
     28 0.587917       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=663 Ack=1129 Win=1020 Len=0
     29 0.591412       10.9.0.1              10.9.0.2              SMB2     322    Create Response File: [Path]\Desktop\qqq
     30 0.591726       10.9.0.2              10.0.0.45             SMB2     370    GetInfo Request FILE_INFO/SMB2_FILE_EA_INFO File: [Path]\Desktop\qqq;GetInfo Request FILE_INFO/SMB2_FILE_STREAM_INFO File: [Path]\Desktop\qqq;GetInfo Request SEC_INFO/SMB2_SEC_INFO_00 File: [Path]\Desktop\qqq
     31 0.634688       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=1394 Ack=1301 Win=1023 Len=0
     32 0.650436       10.0.0.45             10.9.0.2              SMB2     306    GetInfo Response;GetInfo Response;GetInfo Response
     33 0.655687       10.9.0.2              10.9.0.1              SMB2     146    Close Request File: [Path]\Desktop\qqq
     34 0.703847       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=979 Ack=1381 Win=1024 Len=0
     35 0.714000       10.9.0.1              10.9.0.2              SMB2     182    Close Response
     36 0.718687       10.9.0.2              10.0.0.45             SMB2     430    Create Request File: [Path]\Desktop\qqq\testf.csv
     37 0.766287       10.9.0.2              10.9.0.1              TCP      54     5699 → 445 [ACK] Seq=1486 Ack=1429 Win=1023 Len=0
     38 0.777776       10.0.0.45             10.9.0.2              SMB2     410    Create Response File: [Path]\Desktop\qqq\testf.csv
     39 0.778167       10.9.0.2              10.9.0.1              SMB2     370    GetInfo Request FILE_INFO/SMB2_FILE_EA_INFO File: [Path]\Desktop\qqq\testf.csv;GetInfo Request FILE_INFO/SMB2_FILE_STREAM_INFO File: [Path]\Desktop\qqq\testf.csv;GetInfo Request SEC_INFO/SMB2_SEC_INFO_00 File: [Path]\Desktop\qqq\testf.csv
     40 0.819801       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1355 Ack=1737 Win=1023 Len=0
     41 0.837774       10.9.0.1              10.9.0.2              SMB2     346    GetInfo Response;GetInfo Response;GetInfo Response
     42 0.877855       10.9.0.2              10.0.0.45             SMB2     171    Read Request Len:1048576 Off:0 File: [Path]\Desktop\qqq\testf.csv
     43 0.878214       10.9.0.2              10.9.0.1              SMB2     171    Read Request Len:327680 Off:1048576 File: [Path]\Desktop\qqq\testf.csv
     44 0.878234       10.9.0.2              10.9.0.1              SMB2     171    Read Request Len:327680 Off:1376256 File: [Path]\Desktop\qqq\testf.csv
     45 0.878248       10.9.0.2              10.9.0.1              SMB2     171    Read Request Len:65536 Off:2031616 File: [Path]\Desktop\qqq\testf.csv
     46 0.943355       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     47 0.954334       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     48 0.954448       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=4309 Win=1024 Len=0
     49 0.966362       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     50 0.978332       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     51 0.978454       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=6881 Win=1024 Len=0
     52 0.989372       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     53 1.001086       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     54 1.001202       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=9453 Win=1024 Len=0
     55 1.013078       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     56 1.024041       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     57 1.024149       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=12025 Win=1024 Len=0
     58 1.035881       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     59 1.047847       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     60 1.047966       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=14597 Win=1024 Len=0
     61 1.058833       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     62 1.070874       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     63 1.070986       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=17169 Win=1024 Len=0
     64 1.082692       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     65 1.093901       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     66 1.093992       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=19741 Win=1024 Len=0
     67 1.105361       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     68 1.117633       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
     69 1.117756       10.9.0.2              10.0.0.45             TCP      54     5711 → 445 [ACK] Seq=1472 Ack=22313 W
         756 6.037907       10.0.0.45             10.9.0.2              TCP      1340   [TCP Fast Retransmission] [TCP segment of a reassembled PDU]
    757 6.037979       10.9.0.2              10.0.0.45             TCP      82     5711 → 445 [ACK] Seq=1472 Ack=291087 Win=1024 Len=0 SLE=309091 SRE=332239 SLE=302661 SRE=307805 SLE=292373 SRE=301375
    758 6.042192       10.0.0.45             10.9.0.2              TCP      521    [TCP Retransmission] 445 → 5711 [ACK] Seq=291087 Ack=1472 Win=1022 Len=467
    759 6.042254       10.9.0.2              10.0.0.45             TCP      82     5711 → 445 [ACK] Seq=1472 Ack=291554 Win=1022 Len=0 SLE=309091 SRE=332239 SLE=302661 SRE=307805 SLE=292373 SRE=301375
    760 6.048254       10.0.0.45             10.9.0.2              TCP      873    [TCP Retransmission] 445 → 5711 [ACK] Seq=291554 Ack=1472 Win=1022 Len=819
    761 6.048357       10.9.0.2              10.0.0.45             TCP      74     5711 → 445 [ACK] Seq=1472 Ack=301375 Win=1024 Len=0 SLE=309091 SRE=332239 SLE=302661 SRE=307805
    762 6.059069       10.9.0.1              10.9.0.2              TCP      1019   [TCP Retransmission] 445 → 5699 [ACK] Seq=208767 Ack=2504 Win=1020 Len=965
    763 6.059144       10.9.0.2              10.9.0.1              TCP      90     5699 → 445 [ACK] Seq=2504 Ack=209732 Win=1021 Len=0 SLE=240917 SRE=249919 SLE=237059 SRE=239631 SLE=233201 SRE=234487 SLE=226771 SRE=231915
    764 6.070392       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
    765 6.070490       10.9.0.2              10.0.0.45             TCP      74     [TCP Dup ACK 761#1] 5711 → 445 [ACK] Seq=1472 Ack=301375 Win=1024 Len=0 SLE=309091 SRE=333525 SLE=302661 SRE=307805
    766 6.082149       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
    767 6.082239       10.9.0.2              10.0.0.45             TCP      74     [TCP Dup ACK 761#2] 5711 → 445 [ACK] Seq=1472 Ack=301375 Win=1024 Len=0 SLE=309091 SRE=334811 SLE=302661 SRE=307805
    768 6.094102       10.0.0.45             10.9.0.2              TCP      1340   [TCP Fast Retransmission] [TCP segment of a reassembled PDU]
    769 6.094174       10.9.0.2              10.0.0.45             TCP      66     5711 → 445 [ACK] Seq=1472 Ack=307805 Win=1024 Len=0 SLE=309091 SRE=334811
    770 6.105386       10.0.0.45             10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
    771 6.105471       10.9.0.2              10.0.0.45             TCP      66     [TCP Dup ACK 769#1] 5711 → 445 [ACK] Seq=1472 Ack=307805 Win=1024 Len=0 SLE=309091 SRE=336097
    772 6.113050       10.9.0.1              10.9.0.2              TCP      950    [TCP Retransmission] 445 → 5699 [ACK] Seq=209732 Ack=2504 Win=1020 Len=896
    773 6.113118       10.9.0.2              10.9.0.1              TCP      90     5699 → 445 [ACK] Seq=2504 Ack=210628 Win=1024 Len=0 SLE=240917 SRE=249919 SLE=237059 SRE=239631 SLE=233201 SRE=234487 SLE=226771 SRE=231915
    774 6.116711       10.9.0.1              10.9.0.2              TCP      309    [TCP Retransmission] 445 → 5699 [ACK] Seq=210628 Ack=2504 Win=1020 Len=255
    775 6.116774       10.9.0.2              10.9.0.1              TCP      90     5699 → 445 [ACK] Seq=2504 Ack=210883 Win=1023 Len=0 SLE=240917 SRE=249919 SLE=237059 SRE=239631 SLE=233201 SRE=234487 SLE=226771 SRE=231915
    776 6.121453       10.9.0.1              10.9.0.2              TCP      510    [TCP Retransmission] 445 → 5699 [ACK] Seq=210883 Ack=2504 Win=1020 Len=456
    777 6.121521       10.9.0.2              10.9.0.1              TCP      90     5699 → 445 [ACK] Seq=2504 Ack=221627 Win=1024 Len=0 SLE=240917 SRE=249919 SLE=237059 SRE=239631 SLE=233201 SRE=234487 SLE=226771 SRE=231915
    778 6.133876       10.9.0.1              10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]
    779 6.133966       10.9.0.2              10.9.0.1              TCP      90     [TCP Dup ACK 777#1] 5699 → 445 [ACK] Seq=2504 Ack=221627 Win=1024 Len=0 SLE=240917 SRE=251205 SLE=237059 SRE=239631 SLE=233201 SRE=234487 SLE=226771 SRE=231915
    780 6.145584       10.9.0.1              10.9.0.2              TCP      1340   [TCP segment of a reassembled PDU]

Pippin

Ok,

You have in server:

compress lz4-v2
push "compress lz4-v2"

and in client:

comp-lzo

I am not sure this behaves the same between the versions you use, 2.4.8 and 2.5.7.
If possible, update 2.4.8 to latest.
It looks like 2.4.8 does compress and 2.5.7 does not.

You can comment those two #[push] compress and #lzo and try again (meaning without compression).
Take a smaller file, say 20-25 MiB.

Important note regarding compression, see Option: --comp-lzo and Option: --compress:
https://community.openvpn.net/openvpn/wiki/DeprecatedOptions#Option:--comp-lzo
https://community.openvpn.net/openvpn/wiki/VORACLE
If you have many csv (or better said compressible files) then yeah it could be viable to keep using compression, you decide.

Pippin

On irc I was handed this page, take a look at it:
https://community.openvpn.net/openvpn/wiki/Compression#Bottomline

Aitarus

Well, apparently I'm guilty of updating software without prior reading the release notes and such 😛

Your link to site with compression configuration is spot on.
On version >= 2.5 you suppose to add --allow-compression yes and --compress lz4 on server and --allow-compression asym and --compress lz4 on client and it works like a charm 😃
But I noticed that while it works fine on PC client, when you use VPN Connect on Android (in my case), you can connect to server no problem, but there's no traffic. For some reason I had to add "Push compress lz4" on server configuration to get compression properly initialized on Android and only then I could use the tunnel and see PCs behind it.

[timestamp] -- PROTOCOL OPTIONS
       cipher: AES-256-GCM
       digest: NONE
       key-derivation: OpenVPN PRF
       compress: LZ4 init asym = 1
       peer ID:0
       control channel: tls-crypt v2 enabled
[timestamp] -- Lz4 init asym=1
[timestamp] -- EVENT: COMPRESSION_ENABLED   <<== works

But without "push compress lz4", in client logs I had:

[timestamp] -- PROTOCOL OPTIONS
       cipher: AES-256-GCM
       digest: NONE
       key-derivation: OpenVPN PRF
       compress: COMP_STUB			<<== doesn't work
       peer ID:0
       control channel: tls-crypt v2 enabled
[timestamp] -- LZO-ASYM init swap=0 asym=1	<<=\
[timestamp] -- Comp-stub init swap=1		<<== no log entry "EVENT: COMPRESSION_ENABLED" like above

"Push compress lz4" doesn't make any difference on PC client so I guess right now I got what I was looking for. 🙂

Thank you for your help Pippin. 😃

Pippin

Nice, glad you got it solved.