Hello,
I have a problem here that's been bothering me for months and I've run out of ideas.
tl;dr:
The problem is that the transmission is breaking up over an OpenVPN tunnel. It runs perfectly for a week or two and then there are some days when data just doesn't want to flow through the tunnel. The tunnel is still there and according to the log without errors but no data is coming through.
a new establishment of the VPN tunnel works for about 1 minute until the problem occurs again.
* First of all the setup:
┌────────────────────────────────────────────┐
│ VPN-GW-A │
├────────────────────────────────────────────┤
│ Location: │
│ * City A - Location A.1 │
│ │
│ * OS: Debian 9 LTS │
│ * OpenVPN: 2.4.0-6+deb9u3 │
│ │
O---NIC_LAN: │
│ * IP: 192.168.72.1/24 │
│ │
O---NIC_WAN (50 Mbit/10 Mbit): │
│ * IP: 192.168.178.2/24 │
│ * GW: 192.168.178.1 │
│ * DNS: 192.168.178.1 │
│ │
<USERS>-o-----tap0 (VPN): │
│ * openvpn@server.conf │
│ * IP: 10.8.0.1/24 │
│ │
o--o--o-----tap1 (VPN): │
│ │ │ │ * openvpn@vpn-gw-a.conf │
│ │ │ │ * IP: 10.8.1.1/29 │
│ │ │ └────────────────────────────────────────────┘
│ │ │
│ │ │ ┌────────────────────────────────────────────┐
│ │ │ │ VPN-GW-B │
│ │ │ │ External IP: |
│ │ │ ├────────────────────────────────────────────┤
│ │ │ │ Location: │
│ │ │ │ * City A - Location A.2 │
│ │ │ │ │
│ │ │ │ * OS: Debian 10 LTS │
│ │ │ │ * OpenVPN: 2.4.7-1 │
│ │ │ │ │
│ │ │ O---NIC_LAN: │
│ │ │ │ * IP: 192.168.92.1/24 │
│ │ │ │ │
│ │ │ O---NIC_WAN (400 Mbit/50 Mbit): │
│ │ │ │ * IP: DHCP(192.168.1.250/24) │
│ │ │ │ * GW: DHCP │
│ │ │ │ * DNS: DHCP │
│ │ │ │ │
│ │ o-----tap0 (VPN): │
│ │ │ * openvpn-client vpn-gw-b.conf │
│ │ │ * IP: 10.8.1.3/29 │
│ │ └────────────────────────────────────────────┘
│ │
│ │ ┌────────────────────────────────────────────┐
│ │ │ VPN-GW-C │
│ │ │ External IP: |
│ │ ├────────────────────────────────────────────┤
│ │ │ Location: │
│ │ │ * City C - Location C.1 │
│ │ │ │
│ │ │ * OS: Debian 10 LTS │
│ │ │ * OpenVPN: 2.4.7-1 │
│ │ │ │
│ │ O---NIC_LAN: │
│ │ │ * IP: 192.168.82.1/24 │
│ │ │ │
│ │ O---NIC_WAN (100 Mbit/100M bit): │
│ │ │ * IP: DHCP(192.168.1.250/24) │
│ │ │ * GW: 192.168.1.1 │
│ │ │ * DNS: 192.168.1.1 │
│ │ │ │
│ o--------tap0 (VPN): │
│ │ * openvpn-client vpn-gw-c.conf │
│ │ * IP: 10.8.1.2/29 │
│ └────────────────────────────────────────────┘
│
│ ┌────────────────────────────────────────────┐
│ │ VPN-GW-D │
│ │ External IP: |
│ ├────────────────────────────────────────────┤
│ │ Location: │
│ │ * City A - Location A.3 │
│ │ │
│ │ * OS: Debian 10 LTS │
│ │ * OpenVPN: 2.4.7-1 │
│ │ │
│ O---NIC_LAN: │
│ │ * IP: 192.168.102.1/24 │
│ │ │
│ O---NIC_WAN (50 Mbit/10 Mbit) │
│ │ * IP: DHCP(192.168.178.250/24) │
│ │ * GW: 192.168.178.1 │
│ │ * DNS: 192.168.178.1 │
│ │ │
o-----------tap0 (VPN): │
│ * openvpn-client vpn-gw-d.conf │
│ * IP: 10.8.1.4/29 │
└────────────────────────────────────────────┘
The problem occurs only between VPN-GW-A and VPN-GW-C on all other tunnel connections work very well.
About the problem:
If the problem occurs there are the following effects:
* Ping, with default settings, works over the tunnel
* SSH still works, but extremely slow with lags of several seconds to minutes
* Ping and SSH over external IP, so not using the tunnel, work perfectly
* Data transfer like SCP, http(s), RDP etc. do not work at all, and if they do, then at speeds of less than 30 byte/s
Behavior:
* Ping from 10.8.1.1 to 10.8.1.2: running
* start file transfer (10 MiB file with random numbers) from 10.8.1.1 to 10.8.1.2
** transfer starts with 500 KiByte/s and then breaks down to <30Byte/s
** Ping does not get an answer from 10.8.1.2
(The same attempt at the same time over external IPs, without the VPN tunnel, works without problems).
One attempt with ping and tshark:
Ping 10.8.1.2 with a package size of 32 Bytes
• shown here:
◦ An echo request is sent from 10.8.1.1 to 10.8.1.2.
◦ At 10.8.1.2 an echo request arrives and a response is sent to 10.8.1.1.
◦ The response to the echo request arrives at 10.8.1.1.
___@10.8.1.1:~$ ping 10.8.1.2 -s 32
PING 10.8.1.2 (10.8.1.2) 32(60) bytes of data.
40 bytes from 10.8.1.2: icmp_seq=2 ttl=64 time=157 ms
40 bytes from 10.8.1.2: icmp_seq=3 ttl=64 time=313 ms
40 bytes from 10.8.1.2: icmp_seq=4 ttl=64 time=157 ms
40 bytes from 10.8.1.2: icmp_seq=5 ttl=64 time=158 ms
^C
--- 10.8.1.2 ping statistics ---
8 packets transmitted, 4 received, 50% packet loss, time 7074ms
rtt min/avg/max/mdev = 157.109/196.668/313.401/67.397 ms
• tshark log of 10.8.1.1
14 3.330362733 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=3/768, ttl=64 (request in 11)
15 4.018203703 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=4/1024, ttl=64
16 4.175299154 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=4/1024, ttl=64 (request in 15)
17 5.019315863 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=5/1280, ttl=64
18 5.177522183 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=5/1280, ttl=64 (request in 17)
19 6.020508577 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=6/1536, ttl=64
• tshark log of 10.8.1.2
13 3.337046447 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=3/768, ttl=64 (request in 10)
14 4.182078165 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=4/1024, ttl=64
15 4.182125076 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=4/1024, ttl=64 (request in 14)
16 5.184212861 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=5/1280, ttl=64
17 5.184258862 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=5/1280, ttl=64 (request in 16)
18 6.184168968 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=6/1536, ttl=64
19 6.184214117 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=6/1536, ttl=64 (request in 18)
20 8.227912825 10.8.1.1 → 10.8.1.2 ICMP 74 Echo (ping) request id=0x2813, seq=8/2048, ttl=64
21 8.227965086 10.8.1.2 → 10.8.1.1 ICMP 74 Echo (ping) reply id=0x2813, seq=8/2048, ttl=64 (request in 20)
Ping 10.8.1.2 with a package size of 512 Bytes
• shown here:
◦ An echo request is sent from 10.8.1.1 to 10.8.1.2.
◦ At 10.8.1.2 an echo request arrives and a response is sent to 10.8.1.1.
◦ There is no response to the echo request at 10.8.1.1.
___@10.8.1.1:~$ ping 10.8.1.2 -s 512
PING 10.8.1.2 (10.8.1.2) 512(540) bytes of data.
520 bytes from 10.8.1.2: icmp_seq=1 ttl=64 time=156 ms
^C
--- 10.8.1.2 ping statistics ---
8 packets transmitted, 1 received, 87% packet loss, time 7045ms
rtt min/avg/max/mdev = 156.172/156.172/156.172/0.000 ms
• tshark log of 10.8.1.1
309 608.926856464 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=1/256, ttl=64
310 609.082985624 10.8.1.2 → 10.8.1.1 ICMP 554 Echo (ping) reply id=0x1d26, seq=1/256, ttl=64 (request in 309)
311 609.927974348 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=2/512, ttl=64
312 610.948748800 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=3/768, ttl=64
313 611.949725803 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=4/1024, ttl=64
314 612.950469841 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=5/1280, ttl=64
315 613.951221061 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=6/1536, ttl=64
316 614.951202246 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=7/1792, ttl=64
317 615.972732962 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=8/2048, ttl=64
• tshark log of 10.8.1.2
1 0.000000000 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=1/256, ttl=64
2 0.000056733 10.8.1.2 → 10.8.1.1 ICMP 554 Echo (ping) reply id=0x1d26, seq=1/256, ttl=64 (request in 1)
3 7.046158501 10.8.1.1 → 10.8.1.2 ICMP 554 Echo (ping) request id=0x1d26, seq=8/2048, ttl=64
4 7.046205017 10.8.1.2 → 10.8.1.1 ICMP 554 Echo (ping) reply id=0x1d26, seq=8/2048, ttl=64 (request in 3)
As seen in the ping with packet sizes of 512 bytes, the request arrives at 10.8.1.2 and is answered, but these packets do not arrive at 10.8.1.1 anymore.
It doesn't matter which side takes which role.
The packet size where the problem occurs could not be clearly determined. But it is often 256 bytes, sometimes the problem occurs with a packet size of 128 bytes and sometimes 4096 bytes.
* Configuration GW A:
- VPN Configuration ---------------------
@10.8.1.1:# cat /etc/openvpn/vpn-gw-a.conf | grep -P -v '^#|^;'
port 1195
proto udp
dev tap1
ca /etc/openvpn/...
cert /etc/openvpn/...
key /etc/openvpn/...
dh ./.../keys/dh4096.pem
server 10.8.1.0 255.255.255.248
ifconfig-pool-persist ipp-gw-a.txt
client-config-dir ccd-gw-a
route 192.168.82.0 255.255.255.0
route 192.168.92.0 255.255.255.0
keepalive 10 120
user openvpn
group openvpn
persist-key
persist-tun
status openvpn-gw-a-status.log
log-append openvpn-gw-server-a.log
verb 6
- NIC Informations ----------------------
@10.8.1.2:# ip a
tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
link/ether ea:f8:48:4e:35:f8 brd ff:ff:ff:ff:ff:ff
inet 10.8.1.1/29 brd 10.8.1.7 scope global tap1
valid_lft forever preferred_lft forever
* Configuration GW C:
- VPN Configuration ---------------------
@10.8.1.2:# cat /etc/openvpn/vpn-gw-C.conf | grep -P -v '^#|^;'
client
dev tap
proto udp
remote public.ip.of.vpn-gw-a 1195
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/...
cert /etc/openvpn/...
key /etc/openvpn/...
ns-cert-type server
log-append openvpn-gw-C.log
verb 7
route 192.168.72.0 255.255.255.0
- NIC Informations ----------------------
@10.8.1.2:# ip a
tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
link/ether ea:0b:8c:c3:7f:0d brd ff:ff:ff:ff:ff:ff
inet 10.8.1.2/29 brd 10.8.1.7 scope global tap0
valid_lft forever preferred_lft forever
-------------------------------------------
Which has already been tried:
* adjust the MTU -> no improvement
** tun-mtu 1400
** mssfix 1360
* switch to TCP -> generally slow therefore no option
* switch from TAP to TUN -> same problem
* Adjust sndbuf and rcvbuf -> no improvement
* Compression off -> no improvement
The next time the problem occurs, I will provide an extract from the log.
I am open for ideas and further testing and am happy about every suggestion that brings me further.
Thank you very much,
Peter
