Hello guys! I have a problem that I can't figure out on my own. Maybe some of you could help to sort it out for me.
So I have pretty standard client > server setup. Both my client and server are running on Ubuntu 20.04.2 LTS
Client config file is placed at /etc/openvpn/simo_io.conf and looks like this:
client
dev tun
proto udp
remote {{ router_address }} 1194
nobind
user nobody
group nogroup
persist-key
persist-tun
remote-cert-tls server
cipher AES-256-GCM
auth SHA256
verb 3
key-direction 1
keepalive 10 60
ca simo_io.ca
key simo_io.key
cert simo_io.crt
tls-crypt simo_io.ta
After updating /etc/default/openvpn with AUTOSTART="all", things get's up and runs perfectly fine. Although sometimes... like once a month or often client goes in to some weird mode. It starts restarting itself indefinitely and eventually get's killed by systemd with:
Failed with result 'start-limit-hit'
message.
Here is a full events log:
cat /var/log/syslog | grep vpn
......
Nov 4 14:42:37 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: UDP link local: (not bound)
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=6be55771 c9985d06
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: VERIFY KU OK
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Validating certificate extended key usage
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: VERIFY EKU OK
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:37 simo-hub ovpn-simo_io[238569]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:38 simo-hub ovpn-simo_io[238569]: SENT CONTROL [london1]: 'PUSH_REQUEST' (status=1)
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: PUSH: Received control message: 'PUSH_REPLY,route 10.8.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.8.0.18 255.255.0.0,peer-id 0,cipher AES-256-GCM'
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: timers and/or timeouts modified
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: --ifconfig/up options modified
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: route options modified
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: peer-id set
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: adjusting link_mtu to 1624
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: OPTIONS IMPORT: data channel crypto options modified
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: WARNING: Since you are using --dev tun with a point-to-point topology, the second argument to --ifconfig must be an IP address. You are using something (255.255.0.0) that looks more like a netmask. (silence this warning with --ifconfig-nowarn)
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: ROUTE_GATEWAY 192.168.1.254/255.255.255.0 IFACE=enp1s0 HWADDR=84:39:4c:00:49:34
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: TUN/TAP device tun0 opened
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: TUN/TAP TX queue length set to 100
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: /sbin/ip link set dev tun0 up mtu 1500
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: /sbin/ip addr add dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: /sbin/ip route add 10.8.0.1/32 via 255.255.0.0
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: GID set to nogroup
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: UID set to nobody
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Nov 4 14:42:39 simo-hub ovpn-simo_io[238569]: Initialization Sequence Completed
Nov 4 14:42:42 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: cat syslog.1 | grep vpn
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: /sbin/ip route del 10.8.0.1/32
Nov 4 14:42:42 simo-hub openvpn[238609]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: ERROR: Linux route delete command failed: external program exited with error status: 2
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: Closing TUN/TAP interface
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: /sbin/ip addr del dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:42 simo-hub openvpn[238610]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: Linux ip addr del failed: external program exited with error status: 2
Nov 4 14:42:42 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:42 simo-hub ovpn-simo_io[238569]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:42 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: UDP link local: (not bound)
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=f2ef33b0 fce18df8
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: VERIFY KU OK
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Validating certificate extended key usage
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: VERIFY EKU OK
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:42 simo-hub ovpn-simo_io[238615]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:43 simo-hub ovpn-simo_io[238615]: SENT CONTROL [london1]: 'PUSH_REQUEST' (status=1)
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: PUSH: Received control message: 'PUSH_REPLY,route 10.8.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.8.0.18 255.255.0.0,peer-id 1,cipher AES-256-GCM'
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: timers and/or timeouts modified
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: --ifconfig/up options modified
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: route options modified
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: peer-id set
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: adjusting link_mtu to 1624
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: OPTIONS IMPORT: data channel crypto options modified
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: WARNING: Since you are using --dev tun with a point-to-point topology, the second argument to --ifconfig must be an IP address. You are using something (255.255.0.0) that looks more like a netmask. (silence this warning with --ifconfig-nowarn)
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: ROUTE_GATEWAY 192.168.1.254/255.255.255.0 IFACE=enp1s0 HWADDR=84:39:4c:00:49:34
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: TUN/TAP device tun0 opened
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: TUN/TAP TX queue length set to 100
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: /sbin/ip link set dev tun0 up mtu 1500
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: /sbin/ip addr add dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: /sbin/ip route add 10.8.0.1/32 via 255.255.0.0
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: GID set to nogroup
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: UID set to nobody
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Nov 4 14:42:44 simo-hub ovpn-simo_io[238615]: Initialization Sequence Completed
Nov 4 14:42:46 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:46 simo-hub systemd[1]: openvpn.service: Start request repeated too quickly.
Nov 4 14:42:46 simo-hub systemd[1]: openvpn.service: Failed with result 'start-limit-hit'.
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: /sbin/ip route del 10.8.0.1/32
Nov 4 14:42:46 simo-hub openvpn[238662]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: ERROR: Linux route delete command failed: external program exited with error status: 2
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: Closing TUN/TAP interface
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: /sbin/ip addr del dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:46 simo-hub openvpn[238663]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: Linux ip addr del failed: external program exited with error status: 2
Nov 4 14:42:46 simo-hub systemd[1]: openvpn.service: Start request repeated too quickly.
Nov 4 14:42:46 simo-hub systemd[1]: openvpn.service: Failed with result 'start-limit-hit'.
Nov 4 14:42:46 simo-hub ovpn-simo_io[238615]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:46 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: UDP link local: (not bound)
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=4aaf475d a5cadd9a
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: VERIFY KU OK
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Validating certificate extended key usage
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: VERIFY EKU OK
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:46 simo-hub ovpn-simo_io[238668]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:47 simo-hub ovpn-simo_io[238668]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:47 simo-hub ovpn-simo_io[238668]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:47 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: UDP link local: (not bound)
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=374d0e5e 97130754
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: VERIFY KU OK
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Validating certificate extended key usage
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: VERIFY EKU OK
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:47 simo-hub ovpn-simo_io[238676]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238676]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:48 simo-hub ovpn-simo_io[238676]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:48 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: UDP link local: (not bound)
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=b65d58d9 34357907
Nov 4 14:42:48 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:48 simo-hub ovpn-simo_io[238685]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:48 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: UDP link local: (not bound)
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=c84aebd4 fe0d2346
Nov 4 14:42:48 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: VERIFY KU OK
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: Validating certificate extended key usage
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: VERIFY EKU OK
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:48 simo-hub ovpn-simo_io[238690]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:48 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: UDP link local: (not bound)
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=c31a75f2 cb277146
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: VERIFY KU OK
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Validating certificate extended key usage
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: VERIFY EKU OK
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:48 simo-hub ovpn-simo_io[238695]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: SENT CONTROL [london1]: 'PUSH_REQUEST' (status=1)
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: PUSH: Received control message: 'PUSH_REPLY,route 10.8.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.8.0.18 255.255.0.0,peer-id 3,cipher AES-256-GCM'
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: timers and/or timeouts modified
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: --ifconfig/up options modified
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: route options modified
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: peer-id set
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: adjusting link_mtu to 1624
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: OPTIONS IMPORT: data channel crypto options modified
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: WARNING: Since you are using --dev tun with a point-to-point topology, the second argument to --ifconfig must be an IP address. You are using something (255.255.0.0) that looks more like a netmask. (silence this warning with --ifconfig-nowarn)
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: ROUTE_GATEWAY 192.168.1.254/255.255.255.0 IFACE=enp1s0 HWADDR=84:39:4c:00:49:34
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: TUN/TAP device tun0 opened
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: TUN/TAP TX queue length set to 100
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: /sbin/ip link set dev tun0 up mtu 1500
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: /sbin/ip addr add dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: /sbin/ip route add 10.8.0.1/32 via 255.255.0.0
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: GID set to nogroup
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: UID set to nobody
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Nov 4 14:42:49 simo-hub ovpn-simo_io[238695]: Initialization Sequence Completed
Nov 4 14:42:52 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: /sbin/ip route del 10.8.0.1/32
Nov 4 14:42:52 simo-hub openvpn[238732]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: ERROR: Linux route delete command failed: external program exited with error status: 2
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: Closing TUN/TAP interface
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: /sbin/ip addr del dev tun0 local 10.8.0.18 peer 255.255.0.0
Nov 4 14:42:52 simo-hub openvpn[238733]: RTNETLINK answers: Operation not permitted
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: Linux ip addr del failed: external program exited with error status: 2
Nov 4 14:42:52 simo-hub ovpn-simo_io[238695]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:52 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:52 simo-hub systemd[1]: openvpn.service: Succeeded.
Nov 4 14:42:52 simo-hub systemd[1]: openvpn.service: Start request repeated too quickly.
Nov 4 14:42:52 simo-hub systemd[1]: openvpn.service: Failed with result 'start-limit-hit'.
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: WARNING: file 'simo_io.key' is group or others accessible
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: WARNING: file 'simo_io.ta' is group or others accessible
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 27 2021
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: TCP/UDP: Preserving recently used remote address: [AF_INET]46.101.94.145:1194
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: UDP link local: (not bound)
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: UDP link remote: [AF_INET]46.101.94.145:1194
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: TLS: Initial packet from [AF_INET]46.101.94.145:1194, sid=7d5b2bfa 78388ad2
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: VERIFY OK: depth=1, CN=SIMO.io-routers
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: VERIFY KU OK
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Validating certificate extended key usage
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: VERIFY EKU OK
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: VERIFY OK: depth=0, CN=london1
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 384 bit EC, curve: secp384r1
Nov 4 14:42:52 simo-hub ovpn-simo_io[238737]: [london1] Peer Connection Initiated with [AF_INET]46.101.94.145:1194
Nov 4 14:42:53 simo-hub systemd[1]: openvpn.service: Start request repeated too quickly.
Nov 4 14:42:53 simo-hub systemd[1]: openvpn.service: Failed with result 'start-limit-hit'.
Nov 4 14:42:53 simo-hub ovpn-simo_io[238737]: event_wait : Interrupted system call (code=4)
Nov 4 14:42:53 simo-hub ovpn-simo_io[238737]: SIGTERM[hard,] received, process exiting
Nov 4 14:42:53 simo-hub systemd[1]: openvpn@simo_io.service: Succeeded.
Nov 4 14:42:53 simo-hub systemd[1]: openvpn@simo_io.service: Start request repeated too quickly.
Nov 4 14:42:53 simo-hub systemd[1]: openvpn@simo_io.service: Failed with result 'start-limit-hit'.
As you can see these restarts are initiated by
event_wait : Interrupted system call (code=4)
As I understand this suggests that client connection is being terminated by controlling diamond as soon as it successfully establishes connection to the server.
As soon as I realize that something is broken (do not have any automation for that yet, so it happens a few hours or later) i figure out that my openvpn connection is dead by performing [service openvpn status] and all it takes to bring it back is [service openvpn restart]. It comes back and works perfectly fine until it happens again week or two later.
Your help is really appreciated if anyone can suggest anything useful on why it might be happening and how to remediate this.
