I'm having a real headscratcher here.
The setup used to work, but seems to have deteriorated after an upgrade and reboot, possibly.
Server A sits on a fairly sketchy residential ADSL connection. This connection has given us tons of grief in the past. It's very much not business grade, but it's what we have.
Server B sits on a pro connection, where things are much better.
Server A and Server B are connected via SonicWall firewalls, and set up in a VPN (or tunnel, not sure what the SonicWall nomenclature is.)
Server A also runs an OpenVPN server, allowing users to establish a VPN connection from their MacBooks into this. They need to use services on both Server A and B.
If a user establishes a VPN connection to Server A, and works on Server A, everything works fine.
If a user has a shell on Server A and accesses Server B, everything works fine.
If a user establishes a VPN connection and tries to access Server B, things fail, with symptoms that are only compatible with MTU issues. SSH and HTTP connections can be established just fine, but once transfer of larger items starts, the connection halts.
Using ping with the "don't fragment" flag set, and connecting a laptop to Server A via VPN, the following are the largest ping packets that are successful:
Laptop to Server A:
i42@MacBook ~ % ping -c 1 -s 1472 -D server-a.example.local
PING server-a.example.local (192.168.46.10): 1472 data bytes
1480 bytes from 192.168.46.10: icmp_seq=0 ttl=64 time=360.816 ms
--- server-a.example.local ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 360.816/360.816/360.816/0.000 ms
Laptop to Server B:
i42@MacBook ~ % ping -c 1 -s 1410 -D server-b.example.local
PING server-b.example.local (192.168.37.15): 1410 data bytes
1418 bytes from 192.168.37.15: icmp_seq=0 ttl=63 time=728.015 ms
--- server-b.example.local ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 728.015/728.015/728.015/0.000 ms
Server A to Server B:
-bash-4.1$ ping -c 1 -s 1410 -M do server-b.example.local
PING server-b.example.local (192.168.37.15) 1410(1438) bytes of data.
1418 bytes from 192.168.37.15: icmp_seq=1 ttl=64 time=366 ms
--- server-b.example.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 430ms
rtt min/avg/max/mdev = 366.842/366.842/366.842/0.000 ms
The lower MTU of the Server A to Server B link is deliberate, and a result of the crappiness of the ADSL connection Server B sits on.
Server B runs Scientific Linux 6 (I know, I know) fully updated and OpenVPN "OpenVPN 2.4.9 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020".
Here's our server configuration. Note that we use TCP, as UDP suddenly stopped working. We'll try to revert to UDP once we get a service window.
port 1194
proto tcp
dev tun
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/vpnserver.crt
key /etc/openvpn/easy-rsa/keys/vpnserver.key # This file should be kept secret
dh /etc/openvpn/easy-rsa/keys/dh2048.pem
server 10.46.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.46.0 255.255.255.0"
push "route 192.168.37.0 255.255.255.0"
push "dhcp-option DNS 192.168.37.50"
push "dhcp-option DNS 192.168.40.50"
push "dhcp-option DOMAIN example.local"
push "dhcp-option DOMAIN-SEARCH example.local"
duplicate-cn
keepalive 10 120
tls-auth /etc/openvpn/easy-rsa/keys/ta.key 0 # This file is secret
cipher BF-CBC # Blowfish (default)
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append /var/log/openvpn.log
topology subnet
