Page 1 of 1

Client breaks on Cert Read

Posted: Wed May 13, 2020 5:12 am
by superteece
I have two environments: RHEL 8 VMs in ESXi and RHEL 8 VMs on vmware workstation. All running the same version of RHEL, OpenVPN, and OpenSSL.

All was fine until yesterday, now the ESXi based VMs cannot connect to the OpenVPN server. The error is:

Code: Select all

Tue May 12 23:54:16 2020 us=583350 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Tue May 12 23:54:16 2020 us=583495 PKCS#11: pkcs11_initialize - entered
Tue May 12 23:54:16 2020 us=583611 PKCS#11: pkcs11_initialize - return 0-'CKR_OK'
Tue May 12 23:54:16 2020 us=583854 PO_INIT maxevents=4 flags=0x00000002
Tue May 12 23:54:16 2020 us=586535 OpenSSL: error:1418708B:SSL routines:ssl_do_config:unknown command
Tue May 12 23:54:16 2020 us=586615 OpenSSL: error:0909006C:PEM routines:get_name:no start line
Tue May 12 23:54:16 2020 us=586663 Error reading extra certificate
Tue May 12 23:54:16 2020 us=586714 Exiting due to fatal error
The strace shows:

Code: Select all

48577 openat(AT_FDCWD, "bastion2.crt", O_RDONLY) = 3
48577 fstat(3, {st_mode=S_IFREG|0640, st_size=1655, ...}) = 0
48577 read(3, "-----BEGIN CERTIFICATE-----\nMIIE"..., 4096) = 1655
48577 read(3, "", 4096)                 = 0
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5865"..., 102) = 102
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 95) = 95
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 67) = 67
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5867"..., 62) = 62
48577 exit_group(1)                     = ?
48577 +++ exited with 1 +++
I've tried:
  • Reissuing the certs
    Reinstalling OpenVPN
    Reinstalling OpenSSL
    Running OS updates
    Tried the same cert/key combo in the two environments, only breaks in ESXi VMs
    Built a fresh RHEL VM in ESXi from scratch, new cert/key, new conf -- same error
What am I missing?

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 12:00 pm
by TinCanTech
Which version of openvpn ?

viewtopic.php?f=30&t=22603

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 2:30 pm
by superteece
OpenVPN

Code: Select all

OpenVPN 2.4.9 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020
library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=yes enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=yes enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no
OpenSSL

Code: Select all

OpenSSL 1.1.1c FIPS  28 May 2019
RHEL 8

Code: Select all

4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
VMware Workstation

Code: Select all

15.5.2 build-15785246
ESXi
I'm unable to get the version info right now but we just installed it a few months ago so it's very recent if not latest.

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 2:46 pm
by superteece
FIPS mode is off and has never been enabled.

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 3:22 pm
by superteece
Could my issue be at all related to this?

https://patchwork.openvpn.net/patch/1071/

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 3:44 pm
by ecrist
Your best bet is to try downloading and compiling the git tree that contains this patch. Let us know if that fixes your issue.

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 5:25 pm
by superteece
UPDATE
Things we tried since original post:

*Compiled 2.5 from source
*selinux to permissive

With the 2.5 there's a new error when passing all of the needed connection components via CLI:

Code: Select all

[root@bastion2 client]# openvpn --remote <masked> 1194 --tls-client --ca /etc/ipa/ca.crt --cert bastion2.crt --key bastion2.key --tls-auth vi-ta.key --dev tun0
Wed May 13 11:46:09 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Wed May 13 11:46:09 2020 WARNING: file 'vi-ta.key' is group or others accessible
Wed May 13 11:46:09 2020 OpenVPN 2.5_git x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on May 13 2020
Wed May 13 11:46:09 2020 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Wed May 13 11:46:09 2020 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:0E078002:configuration file routines:def_load:system lib
Wed May 13 11:46:09 2020 Warning: TLS client context initialisation has warnings.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib
Wed May 13 11:46:09 2020 Cannot load certificate file bastion2.crt
Wed May 13 11:46:09 2020 Exiting due to fatal error
Additionally, v2.5 cannot read the client.conf file.

So with all the weirdness, I'm re-rolling the VM, going to install OpenVPN3-linux this go around

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 5:42 pm
by TinCanTech
How did you create bastion2.crt ?

Re: Client breaks on Cert Read

Posted: Wed May 13, 2020 11:02 pm
by ecrist
A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

Re: Client breaks on Cert Read

Posted: Wed May 20, 2020 8:29 pm
by marklg
I have a similar error on startup. Openvpn 2.4.9 does not start. Openvpn 2.4.8 starts correctly:

strace from openvpn 2.4.9 failing to start:

Code: Select all

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 108, MSG_NOSIGNAL, NULL, 0) = 108
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 101, MSG_NOSIGNAL, NULL, 0) = 101
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 73, MSG_NOSIGNAL, NULL, 0) = 73
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 68, MSG_NOSIGNAL, NULL, 0) = 68
close(3)                                = 0
write(4, "\1", 1)                       = 1
close(4)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++
strace from openvpn 2.4.8 starting successfully:

Code: Select all

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329709
sendto(3, "<29>May 20 12:44:28 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
close(7)                                = 0
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.key", O_RDONLY) = 7
It then continues successful startup. How can I further troubleshoot this?

This is on RHEL8, fully updated as of today, except for openvpn.

Re: Client breaks on Cert Read

Posted: Sat May 30, 2020 10:42 am
by TinCanTech
@marklg viewtopic.php?f=30&t=22603

It is rude to hijack threads..

Re: Client breaks on Cert Read

Posted: Sat May 30, 2020 3:43 pm
by marklg
I did not see it as hijacking a thread. I saw it as it could possibly be the same issue and pointing to another thread that may help the original poster. I saw that as preferred to wastefully repeating the same information in the other thread.

Regards,

Mark

Re: Client breaks on Cert Read

Posted: Sat May 30, 2020 3:59 pm
by TinCanTech
marklg wrote:
Sat May 30, 2020 3:43 pm
I saw it as it could possibly be the same issue
It is not because as you can see by this comment above:
ecrist wrote:
Wed May 13, 2020 11:02 pm
A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

Re: Client breaks on Cert Read

Posted: Thu Jul 02, 2020 11:39 am
by ccruz
TinCanTech wrote:
Sat May 30, 2020 3:59 pm
marklg wrote:
Sat May 30, 2020 3:43 pm
I saw it as it could possibly be the same issue
It is not because as you can see by this comment above:
ecrist wrote:
Wed May 13, 2020 11:02 pm
A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.
Would you be so kind to provide more details about the fix? I have the same error messages on my Fedora 32 occurring only on 2.4.9, the same issue for all my VPNs. Downgrading to 2.4.8 fixes the issue though.

Re: Client breaks on Cert Read

Posted: Thu Jul 02, 2020 12:17 pm
by TinCanTech
This is not an openvpn issue.
ccruz wrote:
Thu Jul 02, 2020 11:39 am
Would you be so kind to provide more details about the fix?
We don't have a fix.
ccruz wrote:
Thu Jul 02, 2020 11:39 am
I have the same error messages
Then do as instructed here:
TinCanTech wrote:
Sat May 30, 2020 10:42 am
@username viewtopic.php?f=30&t=22603

It is rude to hijack threads..
@Mod .. please close this useless thread.