How to configure MTU correctly?

xand_vladimir · Post by **xand_vladimir** » Fri May 22, 2020 2:29 pm

Hello everyone,

I manage a server farm on Hetzner, one of the component is OpenVPN server.
Since I switched to to vSwitch (service provided by Hetzner) I started suffering poor performance of OpenVPN connections. For example: slow file transfers, constant disconnects, etc.

My first guess is that I should tune the mtu value, however the documentation offered is not clear at all. So, I will put the example and hope that the community could point me in the right direction.

The mtu for network configuration is set to 1400 (this is a requirement by Hetzner). I put an example:

Code: Select all

auto ens19
iface ens19 inet static
      address 5.9.XXX.XXX
      netmask 255.255.255.255
      gateway 5.9.XXX.XXX
      pointopoint 5.9.XXX.XXX
      dns-nameservers 172.16.1.2

auto ens20
iface ens20 inet static
      address 172.16.2.101
      netmask 255.255.255.0 
      mtu 1400

ens19 is the the external network. ens20 is the internal, this same configuration (changing ip address, obviously) are on other machines in the network.

Openvpn config has nothing about mtu configuration, anyway, I put it here:

Server Config

1

local 5.9.XXX.XXX

2

port 443

3

proto tcp

4

topology subnet

5

dev tun0

6

ca /etc/openvpn/server/ca.crt

7

cert /etc/openvpn/server/server.crt

8

key /etc/openvpn/server/server.key

9

dh /etc/openvpn/server/dh.pem

10

server 10.90.0.0 255.255.0.0

11

client-config-dir /etc/openvpn/server/ccd

12

keepalive 600 1800

13

comp-lzo

14

user openvpn

15

group nogroup

16

persist-key

17

persist-tun

18

log-append /var/log/openvpn/openvpn-tcp-443.log

19

verb 6

20

daemon

21

writepid /var/run/openvpn-tcp-443.pid

22

script-security 2

23

learn-address /etc/openvpn/server/learn-address.sh

24

crl-verify /etc/openvpn/server/crl.pem

25

management localhost 7505

So, how exactly should I adjust the MTU?

UPD: Also I noticed that poor performance occurs only when the connection really hits the vSwitch, if the communication occurs between a client and the VM on the same host where OpenVPN Server is deployed everything seems to be fine.

TinCanTech · Post by **TinCanTech** » Fri May 22, 2020 3:04 pm

xand_vladimir wrote: ↑
Fri May 22, 2020 2:29 pm
UPD: Also I noticed that poor performance occurs only when the connection really hits the vSwitch, if the communication occurs between a client and the VM on the same host where OpenVPN Server is deployed everything seems to be fine.

xand_vladimir wrote: ↑
Fri May 22, 2020 2:29 pm
ens19 is the the external network. ens20 is the internal, this same configuration (changing ip address, obviously) are on other machines in the network.

xand_vladimir wrote: ↑

Fri May 22, 2020 2:29 pm

Code: Select all

auto ens19
iface ens19 inet static
      address 5.9.XXX.XXX
      netmask 255.255.255.255
      gateway 5.9.XXX.XXX
      pointopoint 5.9.XXX.XXX
      dns-nameservers 172.16.1.2

auto ens20
iface ens20 inet static
      address 172.16.2.101
      netmask 255.255.255.0 
      mtu 1400

In summary:

The tunnel is between client(general internet) and server(external network)
The tunnel functions perfectly because Hetzner do not interfere with MTU outside of their network
After decrypting tunnel packets then sending the user packets onto Hetzner network problems arise

This has nothing to do with openvpn MTU because the tunnel itself does not traverse Hetzner network.

xand_vladimir · Post by **xand_vladimir** » Fri May 22, 2020 3:48 pm

This has nothing to do with openvpn MTU because the tunnel itself does not traverse Hetzner network.

Seeing it this way yes, it makes sense. However the problem appeared when I started to use vSwitch (mtu 1400), so I thought it is related.

So, what may be the problem then? A real example is:
1. I connect to my OpenVPN server on 5.9.xxx.xxx.
2. I start transfer large file (several Gb) from 172.16.2.xxx.xxx (not OpenVPN host on the vSwitch network) to my host (10.90.xxx.xxx).
3. After, about 40 sec the connection is broken.

Client's log says:

Code: Select all

May 22 17:43:31 crow openvpn[3412569]: Initialization Sequence Completed
May 22 17:44:08 crow openvpn[3412569]: AEAD Decrypt error: cipher final failed
May 22 17:44:08 crow openvpn[3412569]: Fatal decryption error (process_incoming_link), restarting
May 22 17:44:08 crow openvpn[3412569]: TCP/UDP: Closing socket
May 22 17:44:08 crow openvpn[3412569]: /etc/openvpn/scripts/update-systemd-resolved tun0 1500 1555 10.90.0.2 255.255.0.0 restart
May 22 17:44:08 crow openvpn[3412569]: openvpn_execve: unable to fork: Resource temporarily unavailable (errno=11)
May 22 17:44:08 crow openvpn[3412569]: Exiting due to fatal error
May 22 17:44:08 crow openvpn[3412569]: /usr/bin/ip route del 172.16.0.0/16 metric 1
May 22 17:44:08 crow openvpn[3412569]: openvpn_execve: unable to fork: Resource temporarily unavailable (errno=11)
May 22 17:44:08 crow openvpn[3412569]: Exiting due to fatal error

UPD: OK, I was checking the server side log also and that's what I see:

Code: Select all

client-xxx/185.239.201.xxx:46760 MULTI: packet dropped due to output saturation (multi_process_incoming_tun)
client-xxx/185.239.201.xxx:46760 MULTI: packet dropped due to output saturation (multi_process_incoming_tun)
client-xxx/185.239.201.xxx:46760 MULTI: packet dropped due to output saturation (multi_process_incoming_tun)
MULTI: multi_create_instance called
Re-using SSL/TLS context
LZO compression initializing
Control Channel MTU parms [ L:1624 D:1210 EF:40 EB:0 ET:0 EL:3 ]
Data Channel MTU parms [ L:1624 D:1450 EF:124 EB:406 ET:0 EL:3 ]
Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1544,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1544,tun-mtu 1500,proto TCPv

UPD2: It seems like changing tcp-queue-limit parameter to 256 should fix the problem. However, I can't find an explanation why it worked before, when using exactly the same installation.

TinCanTech · Post by **TinCanTech** » Fri May 22, 2020 4:08 pm

xand_vladimir wrote: ↑
Fri May 22, 2020 2:29 pm
The mtu for network configuration is set to 1400 (this is a requirement by Hetzner).

I would want to know why Hetzner insist on interfering with their network at this level.

To me that seems like Hetzner are deliberately making things difficult for their customers,
in order to gouge more money out of them for supporting a problem which Hetzner have
deliberately created.

Also, why are you using --proto tcp ? (It is fraught with technical problems)

xand_vladimir wrote: ↑
Fri May 22, 2020 3:48 pm
I can't find an explanation why it worked before, when using exactly the same installation.

Because it was not exactly the same installation ..

TinCanTech · Post by **TinCanTech** » Fri May 22, 2020 4:35 pm

FYI: If you want to verify the PMTU over which the tunnel does pass use --mtu-test in the client config.

xand_vladimir · Post by **xand_vladimir** » Fri May 22, 2020 4:59 pm

To me that seems like Hetzner are deliberately making things difficult for their customers,

Well, I guess it is like Ryanair, you get what you pay for.

Also, why are you using --proto tcp ? (It is fraught with technical problems)

It was an old requirement for road warriors, you know, 443 port and all this. I fully understand that with OpenVPN the best way to go is via UDP.

FYI: If you want to verify the PMTU over which the tunnel does pass use --mtu-test in the client config.

I am not so expert in networking to fully understand this (it sounds ridiculous, but yes, I am the responsible also for networking in the company). As I may remember, the MTU should never be greater than the network's one. In this case, for example, I guess it should not be greater than 1400 when leaving the OpenVPN server.

TinCanTech · Post by **TinCanTech** » Fri May 22, 2020 5:46 pm

TinCanTech wrote: ↑
Fri May 22, 2020 4:35 pm
FYI: If you want to verify the PMTU over which the tunnel does pass use --mtu-test in the client config.

Infact this is only allowed when openvpn is using UDP.

xand_vladimir wrote: ↑
Fri May 22, 2020 4:59 pm

To me that seems like Hetzner are deliberately making things difficult for their customers,
Well, I guess it is like Ryanair, you get what you pay for.

Pay for something else ...

xand_vladimir wrote: ↑
Fri May 22, 2020 4:59 pm

Also, why are you using --proto tcp ? (It is fraught with technical problems)
It was an old requirement for road warriors, you know, 443 port and all this. I fully understand that with OpenVPN the best way to go is via UDP.

If it was an old requirement is it still required .. ?

xand_vladimir wrote: ↑
Fri May 22, 2020 4:59 pm
I am not so expert in networking to fully understand this (it sounds ridiculous, but yes, I am the responsible also for networking in the company). As I may remember, the MTU should never be greater than the network's one. In this case, for example, I guess it should not be greater than 1400 when leaving the OpenVPN server.

This is simply beyond the scope of Openvpn ..

If all else fails then you can contact me privately: tincanteksup <at> gmail (Fees will apply)

OpenVPN Support Forum

How to configure MTU correctly?

How to configure MTU correctly?

Re: How to configure MTU correctly?

Re: How to configure MTU correctly?

Re: How to configure MTU correctly?

Re: How to configure MTU correctly?

Re: How to configure MTU correctly?

Re: How to configure MTU correctly?