I have two environments: RHEL 8 VMs in ESXi and RHEL 8 VMs on vmware workstation. All running the same version of RHEL, OpenVPN, and OpenSSL. All was fine until yesterday, now the ESXi based VMs cannot connect to the OpenVPN server. The error is: [code]Tue May 12 23:54:16 2020 us=583350 library versions: OpenSSL 1.1.1c FIPS 28 May 2019, LZO 2.08 Tue May 12 23:54:16 2020 us=583495 PKCS#11: pkcs11_initialize - entered Tue May 12 23:54:16 2020 us=583611 PKCS#11: pkcs11_initialize - return 0-'CKR_OK' Tue May 12 23:54:16 2020 us=583854 PO_INIT maxevents=4 flags=0x00000002 Tue May 12 23:54:16 2020 us=586535 OpenSSL: error:1418708B:SSL routines:ssl_do_config:unknown command Tue May 12 23:54:16 2020 us=586615 OpenSSL: error:0909006C:PEM routines:get_name:no start line Tue May 12 23:54:16 2020 us=586663 Error reading extra certificate Tue May 12 23:54:16 2020 us=586714 Exiting due to fatal error[/code] The strace shows: [code]48577 openat(AT_FDCWD, "bastion2.crt", O_RDONLY) = 3 48577 fstat(3, {st_mode=S_IFREG|0640, st_size=1655, ...}) = 0 48577 read(3, "-----BEGIN CERTIFICATE-----\nMIIE"..., 4096) = 1655 48577 read(3, "", 4096) = 0 48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0 48577 write(1, "Tue May 12 23:54:16 2020 us=5865"..., 102) = 102 48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0 48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 95) = 95 48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0 48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 67) = 67 48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0 48577 write(1, "Tue May 12 23:54:16 2020 us=5867"..., 62) = 62 48577 exit_group(1) = ? 48577 +++ exited with 1 +++[/code] I've tried: [list]Reissuing the certs Reinstalling OpenVPN Reinstalling OpenSSL Running OS updates Tried the same cert/key combo in the two environments, only breaks in ESXi VMs Built a fresh RHEL VM in ESXi from scratch, new cert/key, new conf -- same error[/list] What am I missing?

I have a similar error on startup. Openvpn 2.4.9 does not start. Openvpn 2.4.8 starts correctly: strace from openvpn 2.4.9 failing to start: [code]openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7 fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0 read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424 close(7) = 0 getpid() = 329125 sendto(3, "May 20 12:34:04 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7 fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0 read(7, "Certificate:\n Data:\n V"..., 4096) = 4096 read(7, ""..., 4096) = 1474 read(7, "", 4096) = 0 getpid() = 329125 sendto(3, "May 20 12:34:04 openvpn-tun0"..., 108, MSG_NOSIGNAL, NULL, 0) = 108 getpid() = 329125 sendto(3, "May 20 12:34:04 openvpn-tun0"..., 101, MSG_NOSIGNAL, NULL, 0) = 101 getpid() = 329125 sendto(3, "May 20 12:34:04 openvpn-tun0"..., 73, MSG_NOSIGNAL, NULL, 0) = 73 getpid() = 329125 sendto(3, "May 20 12:34:04 openvpn-tun0"..., 68, MSG_NOSIGNAL, NULL, 0) = 68 close(3) = 0 write(4, "\1", 1) = 1 close(4) = 0 exit_group(1) = ? +++ exited with 1 +++ [/code] strace from openvpn 2.4.8 starting successfully: [code]openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7 fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0 read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424 close(7) = 0 getpid() = 329709 sendto(3, "May 20 12:44:28 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7 fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0 read(7, "Certificate:\n Data:\n V"..., 4096) = 4096 read(7, ""..., 4096) = 1474 read(7, "", 4096) = 0 close(7) = 0 openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.key", O_RDONLY) = 7 [/code] It then continues successful startup. How can I further troubleshoot this? This is on RHEL8, fully updated as of today, except for openvpn.

Client breaks on Cert Read

superteece

I have two environments: RHEL 8 VMs in ESXi and RHEL 8 VMs on vmware workstation. All running the same version of RHEL, OpenVPN, and OpenSSL.

All was fine until yesterday, now the ESXi based VMs cannot connect to the OpenVPN server. The error is:

Tue May 12 23:54:16 2020 us=583350 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Tue May 12 23:54:16 2020 us=583495 PKCS#11: pkcs11_initialize - entered
Tue May 12 23:54:16 2020 us=583611 PKCS#11: pkcs11_initialize - return 0-'CKR_OK'
Tue May 12 23:54:16 2020 us=583854 PO_INIT maxevents=4 flags=0x00000002
Tue May 12 23:54:16 2020 us=586535 OpenSSL: error:1418708B:SSL routines:ssl_do_config:unknown command
Tue May 12 23:54:16 2020 us=586615 OpenSSL: error:0909006C:PEM routines:get_name:no start line
Tue May 12 23:54:16 2020 us=586663 Error reading extra certificate
Tue May 12 23:54:16 2020 us=586714 Exiting due to fatal error

The strace shows:

48577 openat(AT_FDCWD, "bastion2.crt", O_RDONLY) = 3
48577 fstat(3, {st_mode=S_IFREG|0640, st_size=1655, ...}) = 0
48577 read(3, "-----BEGIN CERTIFICATE-----\nMIIE"..., 4096) = 1655
48577 read(3, "", 4096)                 = 0
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5865"..., 102) = 102
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 95) = 95
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5866"..., 67) = 67
48577 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1693, ...}) = 0
48577 write(1, "Tue May 12 23:54:16 2020 us=5867"..., 62) = 62
48577 exit_group(1)                     = ?
48577 +++ exited with 1 +++

I've tried:

Reissuing the certs
Reinstalling OpenVPN
Reinstalling OpenSSL
Running OS updates
Tried the same cert/key combo in the two environments, only breaks in ESXi VMs
Built a fresh RHEL VM in ESXi from scratch, new cert/key, new conf -- same error

What am I missing?

TinCanTech

Which version of openvpn ?

https://forums.openvpn.net/viewtopic.php?f=30&t=22603

superteece

OpenVPN

OpenVPN 2.4.9 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020
library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=yes enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=yes enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

OpenSSL

OpenSSL 1.1.1c FIPS  28 May 2019

RHEL 8

4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

VMware Workstation

15.5.2 build-15785246

ESXi
I'm unable to get the version info right now but we just installed it a few months ago so it's very recent if not latest.

superteece

FIPS mode is off and has never been enabled.

superteece

Could my issue be at all related to this?

https://patchwork.openvpn.net/patch/1071/

ecrist

Your best bet is to try downloading and compiling the git tree that contains this patch. Let us know if that fixes your issue.

superteece

UPDATE
Things we tried since original post:

*Compiled 2.5 from source
*selinux to permissive

With the 2.5 there's a new error when passing all of the needed connection components via CLI:

[root@bastion2 client]# openvpn --remote <masked> 1194 --tls-client --ca /etc/ipa/ca.crt --cert bastion2.crt --key bastion2.key --tls-auth vi-ta.key --dev tun0
Wed May 13 11:46:09 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Wed May 13 11:46:09 2020 WARNING: file 'vi-ta.key' is group or others accessible
Wed May 13 11:46:09 2020 OpenVPN 2.5_git x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on May 13 2020
Wed May 13 11:46:09 2020 library versions: OpenSSL 1.1.1c FIPS  28 May 2019, LZO 2.08
Wed May 13 11:46:09 2020 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:0E078002:configuration file routines:def_load:system lib
Wed May 13 11:46:09 2020 Warning: TLS client context initialisation has warnings.
Wed May 13 11:46:09 2020 OpenSSL: error:02001001:system library:fopen:Operation not permitted
Wed May 13 11:46:09 2020 OpenSSL: error:2006D002:BIO routines:BIO_new_file:system lib
Wed May 13 11:46:09 2020 OpenSSL: error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib
Wed May 13 11:46:09 2020 Cannot load certificate file bastion2.crt
Wed May 13 11:46:09 2020 Exiting due to fatal error

Additionally, v2.5 cannot read the client.conf file.

So with all the weirdness, I'm re-rolling the VM, going to install OpenVPN3-linux this go around

TinCanTech

How did you create bastion2.crt ?

ecrist

A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

marklg

I have a similar error on startup. Openvpn 2.4.9 does not start. Openvpn 2.4.8 starts correctly:

strace from openvpn 2.4.9 failing to start:

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 108, MSG_NOSIGNAL, NULL, 0) = 108
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 101, MSG_NOSIGNAL, NULL, 0) = 101
getpid()                                = 329125
sendto(3, "<27>May 20 12:34:04 openvpn-tun0"..., 73, MSG_NOSIGNAL, NULL, 0) = 73
getpid()                                = 329125
sendto(3, "<29>May 20 12:34:04 openvpn-tun0"..., 68, MSG_NOSIGNAL, NULL, 0) = 68
close(3)                                = 0
write(4, "\1", 1)                       = 1
close(4)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

strace from openvpn 2.4.8 starting successfully:

openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/dh2048.pem", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=424, ...}) = 0
read(7, "-----BEGIN DH PARAMETERS-----\nMI"..., 4096) = 424
close(7)                                = 0
getpid()                                = 329709
sendto(3, "<29>May 20 12:44:28 openvpn-tun0"..., 86, MSG_NOSIGNAL, NULL, 0) = 86
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5570, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 4096) = 4096
read(7, "<letters and numbers from certificate >"..., 4096) = 1474
read(7, "", 4096)                       = 0
close(7)                                = 0
openat(AT_FDCWD, "/etc/openvpn/easy-rsa/keys/server.key", O_RDONLY) = 7

It then continues successful startup. How can I further troubleshoot this?

This is on RHEL8, fully updated as of today, except for openvpn.

TinCanTech

@marklg https://forums.openvpn.net/viewtopic.php?f=30&t=22603

It is rude to hijack threads..

marklg

I did not see it as hijacking a thread. I saw it as it could possibly be the same issue and pointing to another thread that may help the original poster. I saw that as preferred to wastefully repeating the same information in the other thread.

Regards,

Mark

TinCanTech

marklg wrote I saw it as it could possibly be the same issue

It is not because as you can see by this comment above:

ecrist wrote A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

ccruz

TinCanTech wrote
marklg wrote I saw it as it could possibly be the same issue
It is not because as you can see by this comment above:
ecrist wrote A point of note here: after dazo and I tried helping this afternoon, it appears the problem actually likes in storage or system permissions. This isn't an OpenVPN issue, and superteece has admitted as much.

Would you be so kind to provide more details about the fix? I have the same error messages on my Fedora 32 occurring only on 2.4.9, the same issue for all my VPNs. Downgrading to 2.4.8 fixes the issue though.

TinCanTech

This is not an openvpn issue.

ccruz wrote Would you be so kind to provide more details about the fix?

We don't have a fix.

ccruz wrote I have the same error messages

Then do as instructed here:

TinCanTech wrote @username https://forums.openvpn.net/viewtopic.php?f=30&t=22603

It is rude to hijack threads..

@Mod .. please close this useless thread.