We renewed a client certificate that was nearing its expiration date.
The new certificate was automatically rolled out to the client via Puppet.
Now the VPN connection fails with the following message appearing in the server log:
Jan 10 12:14:00 s-laugz2ovpn01 ovpn-server[1014]: sarajevo.europe.a-net.de/92.36.213.112:59317 TLS Auth Error: TLS object CN=sarajevo.europe.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth
What do I need to do to tell the server that the cert change is intentional?
I'm afraid your answer is quite too terse for me to make sense of. If you're unwilling or unable to explain yourself further, would you please point me to the part of the OpenVPN documentation I may have overlooked or misunderstood where it explains that OpenVPN works in a way that is so fundamentally incompatible with my rephrased question that you deem an ad hominem attack appropriate?
You have not yet identified the version of openvpn you are using.
Please see: viewtopic.php?f=30&t=22603
You're right. Please pardon my negligence.
OpenVPN Community Edition on Ubuntu Server 20.04LTS
Package version 2.4.7-1ubuntu2.20.04.3
Server Config
port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh4096.pem
server 10.87.0.0 255.255.0.0
topology p2p
push "route 10.103.0.0 255.255.0.0"
push "route 10.102.0.0 255.255.0.0"
... more push route directives ...
client-config-dir ccd
route 10.71.0.0 255.255.0.0
route 10.72.0.0 255.255.0.0
route 10.73.0.0 255.255.0.0
route 10.74.0.0 255.255.0.0
route 10.75.0.0 255.255.0.0
ccd-exclusive
keepalive 60 240
comp-lzo
persist-key
persist-tun
status /var/log/openvpn_status.log
verb 3
crl-verify crl/crl.pem
Client Config
client
dev tun
proto udp
remote vpn.a-net.de 1194
resolv-retry infinite
port 1194
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
comp-lzo
verb 3
Reproduced the error on a test client:
Jan 20 15:39 install renewed certificate via Puppet
Jan 20 15:48 reboot client - connection does not come up
Jan 20 15:56 restart openvpn service on client - connection comes up
Sanitized server log, starting with last successful rekeying before test:
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content)
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) --- /etc/openvpn/client.crt#0112022-01-12 10:34:25.527603036 +0000
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) +++ /tmp/puppet-file20220120-606426-picaz6#0112022-01-20 15:39:25.463916887 +0000
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) @@ -1,42 +1,42 @@
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) -----BEGIN CERTIFICATE-----
{deleted}
Jan 20 15:48:08 testwadiya systemd[1]: Starting OpenVPN service...
Jan 20 15:48:08 testwadiya systemd[1]: Starting OpenVPN connection to client...
Jan 20 15:48:08 testwadiya systemd[1]: Finished OpenVPN service.
Jan 20 15:48:08 testwadiya ovpn-client[835]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jul 19 2021
Jan 20 15:48:08 testwadiya ovpn-client[835]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Jan 20 15:48:08 testwadiya ovpn-client[835]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Jan 20 15:48:08 testwadiya systemd[1]: Started OpenVPN connection to client.
Jan 20 15:48:08 testwadiya ovpn-client[835]: TCP/UDP: Preserving recently used remote address: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 20 15:48:08 testwadiya ovpn-client[835]: UDP link local (bound): [AF_INET][undef]:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: UDP link remote: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: TLS: Initial packet from [AF_INET]ser.ver.ip.addr:1194, sid=e9b9e836 31245c6e
Jan 20 15:48:08 testwadiya ovpn-client[835]: VERIFY OK: depth=1, C=deleted
Jan 20 15:48:08 testwadiya ovpn-client[835]: VERIFY OK: depth=0, C=deleted
Jan 20 15:48:08 testwadiya ovpn-client[835]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:08 testwadiya ovpn-client[835]: [vpn.a-net.de] Peer Connection Initiated with [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:09 testwadiya ovpn-client[835]: SENT CONTROL [vpn.a-net.de]: 'PUSH_REQUEST' (status=1)
Jan 20 15:48:09 testwadiya ovpn-client[835]: AUTH: Received control message: AUTH_FAILED
Jan 20 15:48:09 testwadiya ovpn-client[835]: SIGTERM[soft,auth-failure] received, process exiting
Jan 20 15:48:09 testwadiya systemd[1]: openvpn@client.service: Succeeded.
Jan 20 15:56:47 testwadiya systemd[1]: openvpn.service: Succeeded.
Jan 20 15:56:47 testwadiya systemd[1]: Stopped OpenVPN service.
Jan 20 15:56:47 testwadiya systemd[1]: Stopping OpenVPN service...
Jan 20 15:56:47 testwadiya systemd[1]: Starting OpenVPN service...
Jan 20 15:56:47 testwadiya systemd[1]: Starting OpenVPN connection to client...
Jan 20 15:56:47 testwadiya systemd[1]: Finished OpenVPN service.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jul 19 2021
Jan 20 15:56:47 testwadiya ovpn-client[7674]: library versions: OpenSSL 1.1.1f 31 Mar 2020, LZO 2.10
Jan 20 15:56:47 testwadiya systemd[1]: Started OpenVPN connection to client.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: TCP/UDP: Preserving recently used remote address: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 20 15:56:47 testwadiya ovpn-client[7674]: UDP link local (bound): [AF_INET][undef]:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: UDP link remote: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: TLS: Initial packet from [AF_INET]ser.ver.ip.addr:1194, sid=e4517618 204f4662
Jan 20 15:56:47 testwadiya ovpn-client[7674]: VERIFY OK: depth=1, C=deleted
Jan 20 15:56:47 testwadiya ovpn-client[7674]: VERIFY OK: depth=0, C=deleted
Jan 20 15:56:47 testwadiya ovpn-client[7674]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:56:47 testwadiya ovpn-client[7674]: [vpn.a-net.de] Peer Connection Initiated with [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:48 testwadiya ovpn-client[7674]: SENT CONTROL [vpn.a-net.de]: 'PUSH_REQUEST' (status=1)
Jan 20 15:56:48 testwadiya ovpn-client[7674]: PUSH: Received control message: 'PUSH_REPLY,route 10.103.0.0 255.255.0.0,route 10.102.0.0 255.255.0.0,<deleted>,push-continuation 2'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: PUSH: Received control message: 'PUSH_REPLY,<deleted>,topology p2p,ping 60,ping-restart 240,ifconfig 10.87.72.131 10.87.0.1,peer-id 5,cipher AES-256-GCM,push-continuation 1'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: timers and/or timeouts modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: --ifconfig/up options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: route options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: peer-id set
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: adjusting link_mtu to 1625
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: data channel crypto options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Data Channel: using negotiated cipher 'AES-256-GCM'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:48 testwadiya ovpn-client[7674]: ROUTE_GATEWAY 192.168.123.254/255.255.255.0 IFACE=internet HWADDR=00:26:b9:2d:d0:6d
Jan 20 15:56:48 testwadiya ovpn-client[7674]: TUN/TAP device tun0 opened
Jan 20 15:56:48 testwadiya ovpn-client[7674]: TUN/TAP TX queue length set to 100
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip link set dev tun0 up mtu 1500
Jan 20 15:56:48 testwadiya systemd-udevd[7676]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 20 15:56:48 testwadiya networkd-dispatcher[803]: WARNING:Unknown index 8 seen, reloading interface list
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Link UP
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Gained carrier
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip addr add dev tun0 local 10.87.72.131 peer 10.87.0.1
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Gained IPv6LL
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip route add 10.103.0.0/16 via 10.87.0.1
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip route add 10.102.0.0/16 via 10.87.0.1
{more route add-s deleted}
Jan 20 15:56:48 testwadiya ovpn-client[7674]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Initialization Sequence Completed
Jan 20 15:56:50 testwadiya ntpd[842]: Listen normally on 10 tun0 10.87.72.131:123
Jan 20 15:56:50 testwadiya ntpd[842]: Listen normally on 11 tun0 [fe80::f79c:ed9d:1a62:42ea%8]:123
Jan 20 15:56:50 testwadiya ntpd[842]: 10.87.0.1 local addr 192.168.123.43 -> 10.87.72.131
Jan 20 15:56:50 testwadiya ntpd[842]: new interface(s) found: waking up resolver
Do you know location where client certificate save ? what would client connect if you delete old client certificate and update with new one via Puppet?
restart openvpn service on client - connection comes up it maybe use old certificate so it can connect .it is simple now if you can move old certificate and try push it vie Puppet and post log in here .
what would client connect if you delete old client certificate and update with new one via Puppet?
Sorry, I do not understand that question.
Deleting the old client certificate and replacing it with the new one via Puppet is exactly what I do.
What the client connects to is independent of the certificate.
It is always the same, as determined by the client configuration directive:
it is simple now if you can move old certificate and try push it vie Puppet and post log in here .
I cannot move the old certificate because it is not there anymore.
I have verified with openssl x509 that the certificate in /etc/openvpn/client.crt is the new one.
Also Puppet as a desired state configuration management tool verifies that the certificate in that location is indeed the new one, otherwise it would automatically retransmit it.
I can prove that by manually changing the file in any way, and observing that the next Puppet run overwrites my change and restores the new certificate.
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS Auth Error: TLS object CN=testwadiya.africa.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: move_session: dest=TM_ACTIVE src=TM_UNTRUSTED reinit_src=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: tls_multi_process: untrusted session promoted to semi-trusted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 PUSH: Received control message: 'PUSH_REQUEST'
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Delayed exit in 5 seconds
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'AUTH_FAILED' (status=1)
Jan 20 15:48:15 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SIGTERM[soft,delayed-exit] received, client-instance exiting
If you can fix that** then I will buy you a beer!
** Make Openvpn accept a mid-session reauth certificate change.
What intrigues me is why the server flags this as a "mid-session reauth" in the first place.
Remember, this was the first time the client tried to connect after the reboot.
It didn't have any session yet at this point in time.
Perhaps that's the root cause of the problem: client and server getting out of sync on whether there is a session or not?
Last edited by tgsbn on Fri Jan 21, 2022 12:40 am, edited 1 time in total.
What intrigues me is why the server flags this as a "mid-session reauth".
Remember, this was the first time the client tried to connect after the reboot.
It didn't have any session yet at this point in time.
Perhaps that's the root cause of the problem: client and server getting out of sync on whether there is a session or not?