Client breaks on Cert Read

This forum is for admins who are looking to build or expand their OpenVPN setup.
Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Locked
superteece
OpenVpn Newbie
Posts: 5
Joined: Wed May 13, 2020 4:57 am

Client breaks on Cert Read

Post by superteece » Wed May 13, 2020 5:12 am

I have two environments: RHEL 8 VMs in ESXi and RHEL 8 VMs on vmware workstation. All running the same version of RHEL, OpenVPN, and OpenSSL.

All was fine until yesterday, now the ESXi based VMs cannot connect to the OpenVPN server. The error is:

Code: Select all

Tue May 12 23:54:16 2020 us=583350 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Tue May 12 23:54:16 2020 us=583495 PKCS#11: pkcs11_initialize - entered
Tue May 12 23:54:16 2020 us=583611 PKCS#11: pkcs11_initialize - return 0-'CKR_OK'
Tue May 12 23:54:16 2020 us=583854 PO_INIT maxevents=4 flags=0x00000002
Tue May 12 23:54:16 2020 us=586535 OpenSSL: error:1418708B:SSL routines:ssl_do_config:unknown command
Tue May 12 23:54:16 2020 us=586615 OpenSSL: error:0909006C:PEM routines:get_name:no start line
Tue May 12 23:54:16 2020 us=586663 Error reading extra certificate
Tue May 12 23:54:16 2020 us=586714 Exiting due to fatal error
The strace shows:

Code: Select all

48577 openat(AT_FDCWD, "bastion2.crt", O_RDONLY) = 3
48577 fstat(3, {st_mode=S_IFREG|0640, st_size=1655, ...}) = 0
48577 read(3, "-----BEGIN CERTIFICATE-----\nMIIE"..., 4096) = 1655
48577 read(3, "", 4096)                 = 0
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5865"..., 102) = 102
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 95) = 95
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 67) = 67
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5867"..., 62) = 62
48577 exit_group(1)                     = ?
48577 +++ exited with 1 +++
I've tried:
  • Reissuing the certs
    Reinstalling OpenVPN
    Reinstalling OpenSSL
    Running OS updates
    Tried the same cert/key combo in the two environments, only breaks in ESXi VMs
    Built a fresh RHEL VM in ESXi from scratch, new cert/key, new conf -- same error
What am I missing?

User avatar
TinCanTech
OpenVPN Protagonist
Posts: 7576
Joined: Fri Jun 03, 2016 1:17 pm

Re: Client breaks on Cert Read

Post by TinCanTech » Wed May 13, 2020 12:00 pm

Which version of openvpn ?

viewtopic.php?f=30&t=22603

superteece
OpenVpn Newbie
Posts: 5
Joined: Wed May 13, 2020 4:57 am

Re: Client breaks on Cert Read

Post by superteece » Wed May 13, 2020 2:30 pm

OpenVPN

Code: Select all

OpenVPN 2.4.9 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020
library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=yes enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=yes enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no
OpenSSL

Code: Select all

OpenSSL 1.1.1c FIPS  28 May 2019
RHEL 8

Code: Select all

4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
VMware Workstation

Code: Select all

15.5.2 build-15785246
ESXi
I'm unable to get the version info right now but we just installed it a few months ago so it's very recent if not latest.

superteece
OpenVpn Newbie
Posts: 5
Joined: Wed May 13, 2020 4:57 am

Re: Client breaks on Cert Read

Post by superteece » Wed May 13, 2020 2:46 pm

FIPS mode is off and has never been enabled.

superteece
OpenVpn Newbie
Posts: 5
Joined: Wed May 13, 2020 4:57 am

Re: Client breaks on Cert Read

Post by superteece » Wed May 13, 2020 3:22 pm

Could my issue be at all related to this?

https://patchwork.openvpn.net/patch/1071/

User avatar
ecrist
Forum Team
Posts: 260
Joined: Wed Nov 26, 2008 10:33 pm
Location: Minneapolis, MN
Contact:

Re: Client breaks on Cert Read

Post by ecrist » Wed May 13, 2020 3:44 pm

Your best bet is to try downloading and compiling the git tree that contains this patch. Let us know if that fixes your issue.
OpenVPN Community Administrator
IRC: #openvpn, #openvpn-devel Twitter: @ecrist
Co-Author of Mastering OpenVPN
Author of Troubleshooting OpenVPN

superteece
OpenVpn Newbie
Posts: 5
Joined: Wed May 13, 2020 4:57 am

Re: Client breaks on Cert Read

Post by superteece » Wed May 13, 2020 5:25 pm

UPDATE
Things we tried since original post:

*Compiled 2.5 from source
*selinux to permissive

With the 2.5 there's a new error when passing all of the needed connection components via CLI:

Code: Select all

[root@bastion2 client]# openvpn --remote <masked> 1194 --tls-client --ca /etc/ipa/ca.crt --cert bastion2.crt --key bastion2.key --tls-auth vi-ta.key --dev tun0
Wed May 13 11:46:09 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Wed May 13 11:46:09 2020 WARNING: file 'vi-ta.key' is group or others accessible
Wed May 13 11:46:09 2020 OpenVPN 2.5_git x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on May 13 2020
Wed May 13 11:46:09 2020 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Wed May 13 11:46:09 2020 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:0E078002:configuration file routines:def_load:system lib
Wed May 13 11:46:09 2020 Warning: TLS client context initialisation has warnings.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib
Wed May 13 11:46:09 2020 Cannot load certificate file bastion2.crt
Wed May 13 11:46:09 2020 Exiting due to fatal error
Additionally, v2.5 cannot read the client.conf file.

So with all the weirdness, I'm re-rolling the VM, going to install OpenVPN3-linux this go around

User avatar
TinCanTech
OpenVPN Protagonist
Posts: 7576
Joined: Fri Jun 03, 2016 1:17 pm

Re: Client breaks on Cert Read

Post by TinCanTech » Wed May 13, 2020 5:42 pm

How did you create bastion2.crt ?

User avatar
ecrist
Forum Team
Posts: 260
Joined: Wed Nov 26, 2008 10:33 pm
Location: Minneapolis, MN
Contact:

Re: Client breaks on Cert Read

Post by ecrist » Wed May 13, 2020 11:02 pm

A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.
OpenVPN Community Administrator
IRC: #openvpn, #openvpn-devel Twitter: @ecrist
Co-Author of Mastering OpenVPN
Author of Troubleshooting OpenVPN

marklg
OpenVpn Newbie
Posts: 4
Joined: Mon May 18, 2020 2:40 am

Re: Client breaks on Cert Read

Post by marklg » Wed May 20, 2020 8:29 pm

I have a similar error on startup. Openvpn 2.4.9 does not start. Openvpn 2.4.8 starts correctly:

strace from openvpn 2.4.9 failing to start:

Code: Select all

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 108, MSG_NOSIGNAL, NULL, 0) = 108
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 101, MSG_NOSIGNAL, NULL, 0) = 101
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 73, MSG_NOSIGNAL, NULL, 0) = 73
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 68, MSG_NOSIGNAL, NULL, 0) = 68
close(3)                                = 0
write(4, "\1", 1)                       = 1
close(4)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++
strace from openvpn 2.4.8 starting successfully:

Code: Select all

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329709
sendto(3, "<29>May 20 12:44:28 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
close(7)                                = 0
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.key", O_RDONLY) = 7
It then continues successful startup. How can I further troubleshoot this?

This is on RHEL8, fully updated as of today, except for openvpn.

User avatar
TinCanTech
OpenVPN Protagonist
Posts: 7576
Joined: Fri Jun 03, 2016 1:17 pm

Re: Client breaks on Cert Read

Post by TinCanTech » Sat May 30, 2020 10:42 am

@marklg viewtopic.php?f=30&t=22603

It is rude to hijack threads..

marklg
OpenVpn Newbie
Posts: 4
Joined: Mon May 18, 2020 2:40 am

Re: Client breaks on Cert Read

Post by marklg » Sat May 30, 2020 3:43 pm

I did not see it as hijacking a thread. I saw it as it could possibly be the same issue and pointing to another thread that may help the original poster. I saw that as preferred to wastefully repeating the same information in the other thread.

Regards,

Mark

User avatar
TinCanTech
OpenVPN Protagonist
Posts: 7576
Joined: Fri Jun 03, 2016 1:17 pm

Re: Client breaks on Cert Read

Post by TinCanTech » Sat May 30, 2020 3:59 pm

marklg wrote:
Sat May 30, 2020 3:43 pm
I saw it as it could possibly be the same issue
It is not because as you can see by this comment above:
ecrist wrote:
Wed May 13, 2020 11:02 pm
A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

ccruz
OpenVpn Newbie
Posts: 1
Joined: Thu Jul 02, 2020 11:36 am

Re: Client breaks on Cert Read

Post by ccruz » Thu Jul 02, 2020 11:39 am

TinCanTech wrote:
Sat May 30, 2020 3:59 pm
marklg wrote:
Sat May 30, 2020 3:43 pm
I saw it as it could possibly be the same issue
It is not because as you can see by this comment above:
ecrist wrote:
Wed May 13, 2020 11:02 pm
A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.
Would you be so kind to provide more details about the fix? I have the same error messages on my Fedora 32 occurring only on 2.4.9, the same issue for all my VPNs. Downgrading to 2.4.8 fixes the issue though.

User avatar
TinCanTech
OpenVPN Protagonist
Posts: 7576
Joined: Fri Jun 03, 2016 1:17 pm

Re: Client breaks on Cert Read

Post by TinCanTech » Thu Jul 02, 2020 12:17 pm

This is not an openvpn issue.
ccruz wrote:
Thu Jul 02, 2020 11:39 am
Would you be so kind to provide more details about the fix?
We don't have a fix.
ccruz wrote:
Thu Jul 02, 2020 11:39 am
I have the same error messages
Then do as instructed here:
TinCanTech wrote:
Sat May 30, 2020 10:42 am
@username viewtopic.php?f=30&t=22603

It is rude to hijack threads..
@Mod .. please close this useless thread.

Locked