client certificate renewal -> "client-provided SSL certs unexpectedly changed during mid-session reauth"

tgsbn

We renewed a client certificate that was nearing its expiration date.
The new certificate was automatically rolled out to the client via Puppet.
Now the VPN connection fails with the following message appearing in the server log:

Jan 10 12:14:00 s-laugz2ovpn01 ovpn-server[1014]: sarajevo.europe.a-net.de/92.36.213.112:59317 TLS Auth Error: TLS object CN=sarajevo.europe.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth

What do I need to do to tell the server that the cert change is intentional?

TinCanTech

Try rebooting your client.

tgsbn

Thanks, that helped.

Should we systematically schedule a reboot after client certificate renewals, or is there a milder means to avoid the disconnect?

tgsbn

What should we do to prevent the disruption from reoccurring on future certificate renewals?

TinCanTech

Correct your roll-out process.

tgsbn

I see my original question wasn't sufficiently clear.
Allow me to rephrase it:

What does our rollout process need to do in order to tell the ovpn-server that the cert change is intentional?

TinCanTech

Openvpn does not work that way.

And your roll-out process is badly flawed because you have not taken the time to learn about Openvpn.

tgsbn

I'm afraid your answer is quite too terse for me to make sense of. If you're unwilling or unable to explain yourself further, would you please point me to the part of the OpenVPN documentation I may have overlooked or misunderstood where it explains that OpenVPN works in a way that is so fundamentally incompatible with my rephrased question that you deem an ad hominem attack appropriate?

TinCanTech

It is not an ad hominem attack, it is a verifiable fact.

Moving on ..

You have not yet identified the version of openvpn you are using.
Please see: https://forums.openvpn.net/viewtopic.php?f=30&t=22603

tgsbn wrote would you please point me to the part of the OpenVPN documentation I may have overlooked or misunderstood

That depends on which version you are using, see above.

TinCanTech

I did not know that this was even possible:

ssl_verify.c:1675:            msg(D_TLS_ERRORS, "TLS Auth Error: TLS object CN=%s client-provided SSL certs unexpectedly changed during mid-session reauth",

I have no idea how you triggered it.

TinCanTech

Try this in your client config:

persist-key

tgsbn

TinCanTech wrote You have not yet identified the version of openvpn you are using.
Please see: https://forums.openvpn.net/viewtopic.php?f=30&t=22603

You're right. Please pardon my negligence.

OpenVPN Community Edition on Ubuntu Server 20.04LTS
Package version 2.4.7-1ubuntu2.20.04.3

port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh4096.pem
server 10.87.0.0 255.255.0.0
topology p2p
push "route 10.103.0.0 255.255.0.0"
push "route 10.102.0.0 255.255.0.0"
... more push route directives ...
client-config-dir ccd
route 10.71.0.0 255.255.0.0
route 10.72.0.0 255.255.0.0
route 10.73.0.0 255.255.0.0
route 10.74.0.0 255.255.0.0
route 10.75.0.0 255.255.0.0
ccd-exclusive
keepalive 60 240
comp-lzo
persist-key
persist-tun
status /var/log/openvpn_status.log
verb 3
crl-verify crl/crl.pem

client
dev tun
proto udp
remote vpn.a-net.de 1194
resolv-retry infinite
port 1194
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
comp-lzo
verb 3

Reproduced the error on a test client:
Jan 20 15:39 install renewed certificate via Puppet
Jan 20 15:48 reboot client - connection does not come up
Jan 20 15:56 restart openvpn service on client - connection comes up

Sanitized server log, starting with last successful rekeying before test:

Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: soft reset sec=0 bytes=1645920/-1 pkts=8371/0
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 VERIFY OK: depth=1, C=deleted
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 VERIFY OK: depth=0, C=deleted
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_VER=2.4.7
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_PLAT=linux
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_PROTO=2
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZ4=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZ4v2=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZO=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_COMP_STUB=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_COMP_STUBv2=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_TCPNL=1
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:25:25 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:26:48 s-laugz2ovpn02 openvpn[736]: MULTI: Learn: 10.72.131.1 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 15:27:00 s-laugz2ovpn02 openvpn[736]: MULTI: Learn: 10.72.131.160 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
{more of the same...}
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: new session incoming connection from [AF_INET]cli.ent.ip.addr:1194
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 VERIFY OK: depth=1, C=deleted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 VERIFY OK: depth=0, C=deleted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_VER=2.4.7
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_PLAT=linux
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_PROTO=2
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_NCP=2
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZ4=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZ4v2=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_LZO=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_COMP_STUB=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_COMP_STUBv2=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 peer info: IV_TCPNL=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS Auth Error: TLS object CN=testwadiya.africa.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: move_session: dest=TM_ACTIVE src=TM_UNTRUSTED reinit_src=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: tls_multi_process: untrusted session promoted to semi-trusted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 PUSH: Received control message: 'PUSH_REQUEST'
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Delayed exit in 5 seconds
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'AUTH_FAILED' (status=1)
Jan 20 15:48:15 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SIGTERM[soft,delayed-exit] received, client-instance exiting
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 TLS: Initial packet from [AF_INET]cli.ent.ip.addr:1194, sid=470ecc33 6f9f3de0
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 VERIFY OK: depth=1, C=deleted
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 VERIFY OK: depth=0, C=deleted
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_VER=2.4.7
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_PLAT=linux
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_PROTO=2
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_NCP=2
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_LZ4=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_LZ4v2=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_LZO=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_COMP_STUB=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_COMP_STUBv2=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 peer info: IV_TCPNL=1
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: cli.ent.ip.addr:1194 [testwadiya.africa.a-net.de] Peer Connection Initiated with [AF_INET]cli.ent.ip.addr:1194
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 OPTIONS IMPORT: reading client specific options from: ccd/testwadiya.africa.a-net.de
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 MULTI: Learn: 10.87.72.131 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 MULTI: primary virtual IP for testwadiya.africa.a-net.de/cli.ent.ip.addr:1194: 10.87.72.131
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 MULTI: internal route 10.72.131.0/24 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 15:56:47 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 MULTI: Learn: 10.72.131.0/24 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 PUSH: Received control message: 'PUSH_REQUEST'
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'PUSH_REPLY,route 10.103.0.0 255.255.0.0,route 10.102.0.0 255.255.0.0,<deleted>,push-continuation 2' (status=1)
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'PUSH_REPLY,<deleted>,topology p2p,ping 60,ping-restart 240,ifconfig 10.87.72.131 10.87.0.1,peer-id 5,cipher AES-256-GCM,push-continuation 1' (status=1)
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Data Channel: using negotiated cipher 'AES-256-GCM'
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:48 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:50 s-laugz2ovpn02 openvpn[736]: MULTI: Learn: 10.72.131.1 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 16:00:41 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 MULTI: Learn: 10.72.131.160 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 16:02:38 s-laugz2ovpn02 openvpn[736]: MULTI: Learn: 10.72.131.160 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194
Jan 20 16:04:39 s-laugz2ovpn02 openvpn[736]: MULTI: Learn: 10.72.131.160 -> testwadiya.africa.a-net.de/cli.ent.ip.addr:1194

Sanitized client log, starting with Puppet run:

Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content)
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) --- /etc/openvpn/client.crt#0112022-01-12 10:34:25.527603036 +0000
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) +++ /tmp/puppet-file20220120-606426-picaz6#0112022-01-20 15:39:25.463916887 +0000
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content) @@ -1,42 +1,42 @@
Jan 20 15:39:25 testwadiya puppet-agent[606426]: (/Stage[main]/A_openvpn_20/File[openvpn-client.crt]/content)  -----BEGIN CERTIFICATE-----
{deleted}
Jan 20 15:48:08 testwadiya systemd[1]: Starting OpenVPN service...
Jan 20 15:48:08 testwadiya systemd[1]: Starting OpenVPN connection to client...
Jan 20 15:48:08 testwadiya systemd[1]: Finished OpenVPN service.
Jan 20 15:48:08 testwadiya ovpn-client[835]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jul 19 2021
Jan 20 15:48:08 testwadiya ovpn-client[835]: library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Jan 20 15:48:08 testwadiya ovpn-client[835]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 20 15:48:08 testwadiya systemd[1]: Started OpenVPN connection to client.
Jan 20 15:48:08 testwadiya ovpn-client[835]: TCP/UDP: Preserving recently used remote address: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 20 15:48:08 testwadiya ovpn-client[835]: UDP link local (bound): [AF_INET][undef]:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: UDP link remote: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:08 testwadiya ovpn-client[835]: TLS: Initial packet from [AF_INET]ser.ver.ip.addr:1194, sid=e9b9e836 31245c6e
Jan 20 15:48:08 testwadiya ovpn-client[835]: VERIFY OK: depth=1, C=deleted
Jan 20 15:48:08 testwadiya ovpn-client[835]: VERIFY OK: depth=0, C=deleted
Jan 20 15:48:08 testwadiya ovpn-client[835]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:08 testwadiya ovpn-client[835]: [vpn.a-net.de] Peer Connection Initiated with [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:48:09 testwadiya ovpn-client[835]: SENT CONTROL [vpn.a-net.de]: 'PUSH_REQUEST' (status=1)
Jan 20 15:48:09 testwadiya ovpn-client[835]: AUTH: Received control message: AUTH_FAILED
Jan 20 15:48:09 testwadiya ovpn-client[835]: SIGTERM[soft,auth-failure] received, process exiting
Jan 20 15:48:09 testwadiya systemd[1]: openvpn@client.service: Succeeded.
Jan 20 15:56:47 testwadiya systemd[1]: openvpn.service: Succeeded.
Jan 20 15:56:47 testwadiya systemd[1]: Stopped OpenVPN service.
Jan 20 15:56:47 testwadiya systemd[1]: Stopping OpenVPN service...
Jan 20 15:56:47 testwadiya systemd[1]: Starting OpenVPN service...
Jan 20 15:56:47 testwadiya systemd[1]: Starting OpenVPN connection to client...
Jan 20 15:56:47 testwadiya systemd[1]: Finished OpenVPN service.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jul 19 2021
Jan 20 15:56:47 testwadiya ovpn-client[7674]: library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Jan 20 15:56:47 testwadiya systemd[1]: Started OpenVPN connection to client.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 20 15:56:47 testwadiya ovpn-client[7674]: TCP/UDP: Preserving recently used remote address: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 20 15:56:47 testwadiya ovpn-client[7674]: UDP link local (bound): [AF_INET][undef]:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: UDP link remote: [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:47 testwadiya ovpn-client[7674]: TLS: Initial packet from [AF_INET]ser.ver.ip.addr:1194, sid=e4517618 204f4662
Jan 20 15:56:47 testwadiya ovpn-client[7674]: VERIFY OK: depth=1, C=deleted
Jan 20 15:56:47 testwadiya ovpn-client[7674]: VERIFY OK: depth=0, C=deleted
Jan 20 15:56:47 testwadiya ovpn-client[7674]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:56:47 testwadiya ovpn-client[7674]: [vpn.a-net.de] Peer Connection Initiated with [AF_INET]ser.ver.ip.addr:1194
Jan 20 15:56:48 testwadiya ovpn-client[7674]: SENT CONTROL [vpn.a-net.de]: 'PUSH_REQUEST' (status=1)
Jan 20 15:56:48 testwadiya ovpn-client[7674]: PUSH: Received control message: 'PUSH_REPLY,route 10.103.0.0 255.255.0.0,route 10.102.0.0 255.255.0.0,<deleted>,push-continuation 2'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: PUSH: Received control message: 'PUSH_REPLY,<deleted>,topology p2p,ping 60,ping-restart 240,ifconfig 10.87.72.131 10.87.0.1,peer-id 5,cipher AES-256-GCM,push-continuation 1'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: timers and/or timeouts modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: --ifconfig/up options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: route options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: peer-id set
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: adjusting link_mtu to 1625
Jan 20 15:56:48 testwadiya ovpn-client[7674]: OPTIONS IMPORT: data channel crypto options modified
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Data Channel: using negotiated cipher 'AES-256-GCM'
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 20 15:56:48 testwadiya ovpn-client[7674]: ROUTE_GATEWAY 192.168.123.254/255.255.255.0 IFACE=internet HWADDR=00:26:b9:2d:d0:6d
Jan 20 15:56:48 testwadiya ovpn-client[7674]: TUN/TAP device tun0 opened
Jan 20 15:56:48 testwadiya ovpn-client[7674]: TUN/TAP TX queue length set to 100
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip link set dev tun0 up mtu 1500
Jan 20 15:56:48 testwadiya systemd-udevd[7676]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 20 15:56:48 testwadiya networkd-dispatcher[803]: WARNING:Unknown index 8 seen, reloading interface list
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Link UP
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Gained carrier
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip addr add dev tun0 local 10.87.72.131 peer 10.87.0.1
Jan 20 15:56:48 testwadiya systemd-networkd[751]: tun0: Gained IPv6LL
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip route add 10.103.0.0/16 via 10.87.0.1
Jan 20 15:56:48 testwadiya ovpn-client[7674]: /sbin/ip route add 10.102.0.0/16 via 10.87.0.1
{more route add-s deleted}
Jan 20 15:56:48 testwadiya ovpn-client[7674]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Jan 20 15:56:48 testwadiya ovpn-client[7674]: Initialization Sequence Completed
Jan 20 15:56:50 testwadiya ntpd[842]: Listen normally on 10 tun0 10.87.72.131:123
Jan 20 15:56:50 testwadiya ntpd[842]: Listen normally on 11 tun0 [fe80::f79c:ed9d:1a62:42ea%8]:123
Jan 20 15:56:50 testwadiya ntpd[842]: 10.87.0.1 local addr 192.168.123.43 -> 10.87.72.131
Jan 20 15:56:50 testwadiya ntpd[842]: new interface(s) found: waking up resolver

300000

Do you know location where client certificate save ? what would client connect if you delete old client certificate and update with new one via Puppet?

restart openvpn service on client - connection comes up it maybe use old certificate so it can connect .it is simple now if you can move old certificate and try push it vie Puppet and post log in here .

TinCanTech

@300000

tgsbn wrote Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS Auth Error: TLS object CN=testwadiya.africa.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: move_session: dest=TM_ACTIVE src=TM_UNTRUSTED reinit_src=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: tls_multi_process: untrusted session promoted to semi-trusted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 PUSH: Received control message: 'PUSH_REQUEST'
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Delayed exit in 5 seconds
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'AUTH_FAILED' (status=1)
Jan 20 15:48:15 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SIGTERM[soft,delayed-exit] received, client-instance exiting

If you can fix that** then I will buy you a beer!

** Make Openvpn accept a mid-session reauth certificate change.

300000

mix up old and renew certificate create this trouble. until he delete old certificate and try it again let see how it connect.

TinCanTech

300000 wrote until he delete old certificate and try it again

I am more interested in how to trigger the actual Openvpn message. To my knowledge this should not be possible.

tgsbn

300000 wrote Do you know location where client certificate save ?

Sure: /etc/openvpn/client.crt

300000 wrote what would client connect if you delete old client certificate and update with new one via Puppet?

Sorry, I do not understand that question.
Deleting the old client certificate and replacing it with the new one via Puppet is exactly what I do.
What the client connects to is independent of the certificate.
It is always the same, as determined by the client configuration directive:

remote vpn.a-net.de 1194

300000 wrote restart openvpn service on client - connection comes up it maybe use old certificate so it can connect .

That is not possible.
The old certificate does not exist anywhere on the client anymore after Puppet has deployed the new one.

300000 wrote it is simple now if you can move old certificate and try push it vie Puppet and post log in here .

I cannot move the old certificate because it is not there anymore.
I have verified with openssl x509 that the certificate in /etc/openvpn/client.crt is the new one.
Also Puppet as a desired state configuration management tool verifies that the certificate in that location is indeed the new one, otherwise it would automatically retransmit it.
I can prove that by manually changing the file in any way, and observing that the next Puppet run overwrites my change and restores the new certificate.

TinCanTech

@tgsbn Can you remove persist-tun from one of your clients for testing purposes ? Keep persist-key

Also, confirm if the same error occurs on the server for that client:

TLS Auth Error: TLS object CN=testwadiya.africa.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth

tgsbn

TinCanTech wrote
tgsbn wrote Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS Auth Error: TLS object CN=testwadiya.africa.a-net.de client-provided SSL certs unexpectedly changed during mid-session reauth
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: move_session: dest=TM_ACTIVE src=TM_UNTRUSTED reinit_src=1
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 TLS: tls_multi_process: untrusted session promoted to semi-trusted
Jan 20 15:48:08 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 PUSH: Received control message: 'PUSH_REQUEST'
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 Delayed exit in 5 seconds
Jan 20 15:48:09 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SENT CONTROL [testwadiya.africa.a-net.de]: 'AUTH_FAILED' (status=1)
Jan 20 15:48:15 s-laugz2ovpn02 openvpn[736]: testwadiya.africa.a-net.de/cli.ent.ip.addr:1194 SIGTERM[soft,delayed-exit] received, client-instance exiting
If you can fix that** then I will buy you a beer!

** Make Openvpn accept a mid-session reauth certificate change.

What intrigues me is why the server flags this as a "mid-session reauth" in the first place.
Remember, this was the first time the client tried to connect after the reboot.
It didn't have any session yet at this point in time.

Perhaps that's the root cause of the problem: client and server getting out of sync on whether there is a session or not?

TinCanTech

tgsbn wrote What intrigues me is why the server flags this as a "mid-session reauth".
Remember, this was the first time the client tried to connect after the reboot.
It didn't have any session yet at this point in time.

Perhaps that's the root cause of the problem: client and server getting out of sync on whether there is a session or not?

Exactly.

You could use --explicit-exit-notify N