Failing connections after moving openvpn config to new server

[kandresen]

I have moved my OpenVPN installation from an old Debian 7 jessie server to a new Debian stretch server without updating neither certificates, nor configuration. The only line in my client configuration that have changed is the new servers address. Initially it looked like everything is working, I can connect all the clients to the server, but they time out, and never recover. Before they would reconnect, but now I get fatal errors, and I need to manually restart the openvpn on the clients to recover.

Both the old server and new server used their respective standard OpenVPN packages from the Debian repository. So I guess there must be some change to parameters I need on either client or server. This is my server setup:

Code: Select all

port 1194
dev tun
proto udp
user nobody
group nogroup
persist-key
persist-tun
mute-replay-warnings
status /var/log/openvpn/status.log
log-append /var/log/openvpn/openvpn.log
ca ca.crt
cert server.crt
key server.key
dh dh2048.pem
server 10.15.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
client-to-client
keepalive 10 120
tls-auth ta.key 0
cipher AES-128-CBC
comp-lzo
verb 4
mute 20

This is my Ubuntu 16.04 client configuration:

Code: Select all

client
dev tun
proto udp
remote myserver.com 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
log-append /var/log/openvpn/myserver.log
mute-replay-warnings
ca ca.crt
cert client.crt
key client.key
tls-auth ta.key 1
;remote-cert-tls server
ns-cert-type server
cipher AES-128-CBC
comp-lzo
verb 3
mute 20

From server log:

Code: Select all

Wed Aug  9 14:14:40 2017 us=525289 265 variation(s) on previous 20 message(s) suppressed by --mute
Wed Aug  9 14:14:40 2017 us=525297 OpenVPN 2.4.0 [git:master/d73f7253d939e293+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 22 2017
Wed Aug  9 14:14:40 2017 us=525316 library versions: OpenSSL 1.0.2l  25 May 2017, LZO 2.08
Wed Aug  9 14:14:40 2017 us=539472 Diffie-Hellman initialized with 2048 bit key
Wed Aug  9 14:14:40 2017 us=540384 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Aug  9 14:14:40 2017 us=540404 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
...
Wed Aug  9 20:50:19 2017 us=116762 client/<my ip>:41968 SENT CONTROL [client]: 'PUSH_REPLY,route 10.15.0.0 255.255.255.0,topology net30,ping 10,ping-restart 120,ifconfig 10.15.0.6 10.15.0.5,peer-id 0' (status=1)
Wed Aug  9 20:54:19 2017 us=668595 client/<my ip>:41968 [kenneth] Inactivity timeout (--ping-restart), restarting
Wed Aug  9 20:54:19 2017 us=668695 client/<my ip>:41968 SIGUSR1[soft,ping-restart] received, client-instance restarting

From client.log:

Code: Select all

Wed Aug  9 14:23:17 2017 OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jun 22 2017
Wed Aug  9 14:23:17 2017 library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.08
Wed Aug  9 14:23:17 2017 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Wed Aug  9 14:23:17 2017 Control Channel Authentication: using 'ta.key' as a OpenVPN static key file
Wed Aug  9 14:23:17 2017 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Aug  9 14:23:17 2017 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Aug  9 14:23:17 2017 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed Aug  9 14:23:18 2017 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Wed Aug  9 14:23:18 2017 UDPv4 link local: [undef]
Wed Aug  9 14:23:18 2017 UDPv4 link remote: [AF_INET]<server ip>:1194
Wed Aug  9 14:23:19 2017 TLS: Initial packet from [AF_INET]<server ip>:1194, sid=610f0042 301cf4cb
...
Wed Aug  9 20:49:43 2017 35 variation(s) on previous 20 message(s) suppressed by --mute
Wed Aug  9 20:49:43 2017 [server] Inactivity timeout (--ping-restart), restarting
Wed Aug  9 20:49:43 2017 SIGUSR1[soft,ping-restart] received, process restarting
Wed Aug  9 20:49:43 2017 Restart pause, 2 second(s)
Wed Aug  9 20:49:45 2017 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Wed Aug  9 20:49:45 2017 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed Aug  9 20:49:45 2017 UDPv4 link local: [undef]
Wed Aug  9 20:49:45 2017 UDPv4 link remote: [AF_INET]<server ip>:1194
Wed Aug  9 20:49:45 2017 TLS: Initial packet from [AF_INET]<server ip>:1194, sid=83469552 01a6ca99
Wed Aug  9 20:49:48 2017 PUSH: Received control message: 'PUSH_REPLY,route 10.15.0.0 255.255.255.0,topology net30,ping 10,ping-restart 120,ifconfig 10.15.0.6 10.15.0.5,peer-id 0'
Wed Aug  9 20:49:48 2017 OPTIONS IMPORT: timers and/or timeouts modified
Wed Aug  9 20:49:48 2017 OPTIONS IMPORT: --ifconfig/up options modified
Wed Aug  9 20:49:48 2017 OPTIONS IMPORT: route options modified
Wed Aug  9 20:49:48 2017 OPTIONS IMPORT: peer-id set
Wed Aug  9 20:49:48 2017 OPTIONS IMPORT: adjusting link_mtu to 1561
Wed Aug  9 20:49:48 2017 Preserving previous TUN/TAP instance: tun1
Wed Aug  9 20:49:48 2017 NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Wed Aug  9 20:49:48 2017 /sbin/ip route del 10.15.0.0/24
RTNETLINK answers: Operation not permitted
Wed Aug  9 20:49:48 2017 ERROR: Linux route delete command failed: external program exited with error status: 2
Wed Aug  9 20:49:48 2017 Closing TUN/TAP interface
Wed Aug  9 20:49:48 2017 /sbin/ip addr del dev tun1 local 10.15.0.6 peer 10.15.0.5
RTNETLINK answers: Operation not permitted
Wed Aug  9 20:49:48 2017 Linux ip addr del failed: external program exited with error status: 2
Wed Aug  9 20:49:48 2017 /etc/openvpn/update-resolv-conf tun1 1500 1561 10.15.0.6 10.15.0.5 init
Wed Aug  9 20:49:49 2017 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=wlp3s0 HWADDR=5c:51:4f:2b:e6:16
Wed Aug  9 20:49:49 2017 ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
Wed Aug  9 20:49:49 2017 Exiting due to fatal error

Notice - all the clients are running the same ubuntu 16.04 version. None crashed with the previous server, so why are these clients crashing now? Unfortunately the old server is no more, so I cannot return to test.
In the meantime, I have created a cronjob on the clients to check if openvpn is running, and restarting the service if not. It seems like the problem only occur on the clients if no data is transmitted for some time.

[kandresen]

I have discovered that all these client systems have bridge setup where ipv6 is disabled on the bridge.sysctl -a|grep disable_ipv6
sysctl: reading key "net.ipv6.conf.all.stable_secret"
net.ipv6.conf.all.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.br1.stable_secret"
net.ipv6.conf.br1.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.default.stable_secret"
net.ipv6.conf.default.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.eno1.stable_secret"
net.ipv6.conf.eno1.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
net.ipv6.conf.lo.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.tun0.stable_secret"
net.ipv6.conf.tun0.disable_ipv6 = 0
sysctl: reading key "net.ipv6.conf.virbr0.stable_secret"
net.ipv6.conf.virbr0.disable_ipv6 = 1
sysctl: reading key "net.ipv6.conf.virbr0-nic.stable_secret"
net.ipv6.conf.virbr0-nic.disable_ipv6 = 0

I cannot find any immediate reason for why the bridges have ipv6 disabled, but since this is the case, I guess the error I have might be related with this: https://community.openvpn.net/openvpn/ticket/849

[kandresen]

The disabled IPv6 on virbr0 is most likely not the problem in this case. I am realizing I added the ipv6 result from a different client than that of the client log earlier. Sorry about that. There are two bridges on that client: "br1" which I created in /etc/network/interfaces and which act as the "default gateway", the other being "virbr0" which was auto-created by virt-manager but is not in use as I configured virt-manager to use br1 instead.
The OpenVPN tunnel is tun0, and must have been established on top of br1.
The interesting question is why virt-manager created a bridge with ipv6 disabled since it apparently have no problem using the ipv6 enabled bridge anyway. Regardless, despite a ipv6 disabled bridge existed on my client system, it is most likely not the reason for OpenVPN crashing in my case, I will simply remove this device entirely as there is nothing using it.

[kandresen]

Digression: IPv6 appear to have been disabled by default in libvirt related with Redhat patches from 2009 which also got added to all other distros:
Bug 501934:"libvirt bridge should have IPv6 disabled" Ref: <https://bugzilla.redhat.com/show_bug.cgi?id=501934>

Despite expecting the error to reappear, I will leave the system as is for now as none of the OpenVPN clients have new error messages since yesterday after removing the default bridge created by libvirt.