Server certificate expired but not really

[gozer]

We are connecting to AWS using a client VPN connection from 2 different on prem machines. This connection has been working for a while. We started getting the server certificate expired messages below (possibly) after a patch and reboot done by the ops team. I was on the phone with AWS support last night and we triple verified the key, cert and ca were all valid.

While talking to AWS support, we tweaked the config adding unique random string in front of the DNS names on each machine (per AWS install instructions). We tried quite a few things on the AWS side to no avail. Now one of the machines (lower lanes) is working fine but the production machine loops through the reconnect messages spitting out the certificate has expired message evert 300 seconds.

But... The connection is established and works! I can SSH to AWS from the client and it connects and stays connected while the ovpn logs shows a restart every 300 seconds. The core issue is that the performance of the connection is degraded; we send Kafka messages and are seeing longer lag times.

How can we troubleshoot this problem?

Startup log:

Code: Select all

-- Logs begin at Thu 2023-01-26 07:51:56 CST, end at Thu 2023-01-26 07:58:39 CST. --
Jan 26 07:52:06 REDACTED openvpn[1132]: Thu Jan 26 07:52:06 2023 OpenVPN 2.4.12 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO
Jan 26 07:52:06 REDACTED openvpn[1132]: Thu Jan 26 07:52:06 2023 library versions: OpenSSL 1.0.2k-fips  26 Jan 2017, LZO 2.06
Jan 26 07:52:06 REDACTED openvpn[1132]: Thu Jan 26 07:52:06 2023 WARNING: Your certificate has expired!
Jan 26 07:52:06 REDACTED openvpn[1149]: Thu Jan 26 07:52:06 2023 OpenVPN 2.4.12 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO
Jan 26 07:52:06 REDACTED openvpn[1149]: Thu Jan 26 07:52:06 2023 library versions: OpenSSL 1.0.2k-fips  26 Jan 2017, LZO 2.06
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]IP-REDACTED:443
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 UDP link local: (not bound)
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 UDP link remote: [AF_INET]IP-REDACTED:443
Jan 26 07:52:07 REDACTED openvpn[1149]: Thu Jan 26 07:52:07 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]IP-REDACTED:443
Jan 26 07:52:07 REDACTED openvpn[1149]: Thu Jan 26 07:52:07 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 26 07:52:07 REDACTED openvpn[1149]: Thu Jan 26 07:52:07 2023 UDP link local: (not bound)
Jan 26 07:52:07 REDACTED openvpn[1149]: Thu Jan 26 07:52:07 2023 UDP link remote: [AF_INET]IP-REDACTED:443
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 TLS: Initial packet from [AF_INET]IP-REDACTED:443, sid=9554ff5e e29f38fe
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 VERIFY OK: depth=1, CN=CN-REDACTED
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 VERIFY ERROR: depth=0, error=certificate has expired: CN=server, serial=245904959834434997247351322248653797910
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 OpenSSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 TLS_ERROR: BIO read tls_read_plaintext error
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 TLS Error: TLS object -> incoming plaintext read error
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 TLS Error: TLS handshake failed
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 SIGUSR1[soft,tls-error] received, process restarting
Jan 26 07:52:07 REDACTED openvpn[1132]: Thu Jan 26 07:52:07 2023 Restart pause, 5 second(s)

This repeats every 5 minutes:

Code: Select all

Jan 26 09:49:45 REDACTED openvpn[1132]: Thu Jan 26 09:49:45 2023 Restart pause, 300 second(s)
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]REDACTED:443
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 UDP link local: (not bound)
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 UDP link remote: [AF_INET]REDACTED:443
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 TLS: Initial packet from [AF_INET]REDACTED:443, sid=5b37cb01 1a8239a9
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 VERIFY OK: depth=1, CN=REDACTED
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 VERIFY ERROR: depth=0, error=certificate has expired: CN=server, serial=245904959834434997247351322248653797910
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 OpenSSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 TLS_ERROR: BIO read tls_read_plaintext error
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 TLS Error: TLS object -> incoming plaintext read error
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 TLS Error: TLS handshake failed
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 SIGUSR1[soft,tls-error] received, process restarting
Jan 26 09:54:45 REDACTED openvpn[1132]: Thu Jan 26 09:54:45 2023 Restart pause, 300 second(s)
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]REDACTED:443
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 UDP link local: (not bound)
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 UDP link remote: [AF_INET]REDACTED:443
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 TLS: Initial packet from [AF_INET]REDACTED:443, sid=332d0106 cd4409fb
Jan 26 09:59:45 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 VERIFY OK: depth=1, CN=Levis
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 VERIFY ERROR: depth=0, error=certificate has expired: CN=server, serial=245904959834434997247351322248653797910
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 OpenSSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 TLS_ERROR: BIO read tls_read_plaintext error
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 TLS Error: TLS object -> incoming plaintext read error
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 TLS Error: TLS handshake failed
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 SIGUSR1[soft,tls-error] received, process restarting
Jan 26 09:59:46 REDACTED openvpn[1132]: Thu Jan 26 09:59:45 2023 Restart pause, 300 second(s)

[300000]

I think it use catch old certificate on memory so it can connect but if you restart system it will not connect anymorre

There is CA certificate. Server certificate. So do you know how to check CA certificate and server certificate ? This is live production so just carefully or everything will stop. You need to have back up second link system to keep product line working so you can changes it .