Community, Twice now we have had openvpn go down on a CentOS 6.5 host. It looks like a potentially thorny problem to debug. The problem seems to coincide with a scheduled CRL verification process. Here is what that process and the subsequent crash look like (some [info] removed): [code]Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 CRL CHECK OK: C=[...], ST=[...], L=[...], O=[...], OU=[...], CN=[...], emailAddress=[...] Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 VERIFY OK: depth=1, C=[...], ST=[...], L=[...], O=[...], OU=[...], CN=[...], emailAddress=[...] Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 CRL CHECK OK: C=[...], ST=[...], O=[...], OU=[...], CN=[...], emailAddress=[...] Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 VERIFY OK: depth=0, C=[...], ST=[...], O=[...], OU=[...], CN=[...], emailAddress=[...] Jan 22 06:27:29 [vpnserver] abrt[23112]: File '/usr/sbin/openvpn' seems to be deleted Jan 22 06:27:29 [vpnserver] abrtd: Directory 'ccpp-2014-01-22-06:27:29-1524' creation detected Jan 22 06:27:29 [vpnserver] abrt[23112]: Saved core dump of pid 1524 (/usr/sbin/openvpn) to /var/spool/abrt/ccpp-2014-01-22-06:27:29-1524 (1691648 bytes) Jan 22 06:27:30 [vpnserver] abrtd: Package 'openvpn' isn't signed with proper key Jan 22 06:27:30 [vpnserver] abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-01-22-06:27:29-1524' exited with 1 Jan 22 06:27:30 [vpnserver] abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-01-22-06:27:29-1524' [/code] Sad that the core dump isn't preserved by abrtd...I might be able to figure out a way to retain it. Oddly, attempts to restart openvpn after this also fail: [code]Jan 22 11:17:57 [vpnserver] openvpn[23748]: OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013 Jan 22 11:17:57 [vpnserver] openvpn[23748]: PLUGIN_INIT: POST /usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so '[/usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so] [login]' intercepted=PLUGIN_AUTH_USER_PASS_VERIFY Jan 22 11:17:58 [vpnserver] openvpn[23748]: Diffie-Hellman initialized with 2048 bit key Jan 22 11:17:58 [vpnserver] openvpn[23748]: Socket Buffers: R=[124928->131072] S=[124928->131072] Jan 22 11:17:58 [vpnserver] openvpn[23748]: TCP/UDP: Socket bind failed on local address [undef]: Address already in use Jan 22 11:17:58 [vpnserver] openvpn[23748]: Exiting due to fatal error [/code] And rebooting the server then has an issue as well: the shutdown hangs after "iptables: Unloading modules: " is displayed. I can reboot normally when openvpn doesn't have this issue. After resetting the system, it reboots and everything works fine again. For now I'm monitoring the port to detect the failure so I can bring it back up again quickly...but a way to resolve the problem would be very valuable.

OpenVPN 2.3.2 stability issue on CentOS 6.5

symphero

Community,

Twice now we have had openvpn go down on a CentOS 6.5 host. It looks like a potentially thorny problem to debug.

The problem seems to coincide with a scheduled CRL verification process.

Here is what that process and the subsequent crash look like (some [info] removed):

Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 CRL CHECK OK: C=[...], ST=[...], L=[...], O=[...], OU=[...], CN=[...], emailAddress=[...]
Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 VERIFY OK: depth=1, C=[...], ST=[...], L=[...], O=[...], OU=[...], CN=[...], emailAddress=[...]
Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 CRL CHECK OK: C=[...], ST=[...], O=[...], OU=[...], CN=[...], emailAddress=[...]
Jan 22 06:27:29 [vpnserver] openvpn[1532]: [User]/[Host]:55835 VERIFY OK: depth=0, C=[...], ST=[...], O=[...], OU=[...], CN=[...], emailAddress=[...]
Jan 22 06:27:29 [vpnserver] abrt[23112]: File '/usr/sbin/openvpn' seems to be deleted
Jan 22 06:27:29 [vpnserver] abrtd: Directory 'ccpp-2014-01-22-06:27:29-1524' creation detected
Jan 22 06:27:29 [vpnserver] abrt[23112]: Saved core dump of pid 1524 (/usr/sbin/openvpn) to /var/spool/abrt/ccpp-2014-01-22-06:27:29-1524 (1691648 bytes)
Jan 22 06:27:30 [vpnserver] abrtd: Package 'openvpn' isn't signed with proper key
Jan 22 06:27:30 [vpnserver] abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-01-22-06:27:29-1524' exited with 1
Jan 22 06:27:30 [vpnserver] abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-01-22-06:27:29-1524'

Sad that the core dump isn't preserved by abrtd...I might be able to figure out a way to retain it.

Oddly, attempts to restart openvpn after this also fail:

Jan 22 11:17:57 [vpnserver] openvpn[23748]: OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013
Jan 22 11:17:57 [vpnserver] openvpn[23748]: PLUGIN_INIT: POST /usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so '[/usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so] [login]' intercepted=PLUGIN_AUTH_USER_PASS_VERIFY
Jan 22 11:17:58 [vpnserver] openvpn[23748]: Diffie-Hellman initialized with 2048 bit key
Jan 22 11:17:58 [vpnserver] openvpn[23748]: Socket Buffers: R=[124928->131072] S=[124928->131072]
Jan 22 11:17:58 [vpnserver] openvpn[23748]: TCP/UDP: Socket bind failed on local address [undef]: Address already in use
Jan 22 11:17:58 [vpnserver] openvpn[23748]: Exiting due to fatal error

And rebooting the server then has an issue as well: the shutdown hangs after "iptables: Unloading modules: " is displayed. I can reboot normally when openvpn doesn't have this issue.

After resetting the system, it reboots and everything works fine again.

For now I'm monitoring the port to detect the failure so I can bring it back up again quickly...but a way to resolve the problem would be very valuable.

rainbow6

i have 25 servers running centos 6.5 and have never seen or encountered this error.
Is this standard openvpn binaries?
How did you start your openvpn?
And how did you update your crl?

Based on the log, your openvpn is still running when your start your openvpn. use
service openvpn restart
or
issue a killall openvpn and then start by using service or /etc/init.d

Disable iptables and iptables 6 in chkconfig or setup

symphero

I realize OpenVPN is a stable, solid product...that's why I'm especially baffled by this behavior.

I'm using the standard EPEL OpenVPN package, and it's started through the normal init script.

The few times I've revoked a certificate, it's been on one of our PKI management hosts (also running CentOS 6.5), using the command

# openssl ca -revoke <cert>

I've been using openssl directly, rather than using the popular wrapper scripts.

I'll see if I can make a better recovery by cleaning up the remnants of the daemon process...but I'm not free to shut off iptables, which is a necessary security control in this environment (and a practical necessity as well, since access to the internal network requires NAT).

I've added the EPEL GPG key to the abrt keys list, so I should get a clean core dump if there is another recurrence.

symphero

OK, another crash today, and abrt succeeded in generating crash info.

It looks like our use of PAM authentication is triggering a SIGABRT in some shared libraries, but I can't tell what the root cause is without a full debug environment for OpenVPN. I may disable our local PAM authentication and rely on certs alone for authentication if this continues to be an issue.

Here's a portion of the backtrace from the coredump report:

[New Thread 1422]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/openvpn --daemon --writepid /var/run/openvpn/qamcpdvpn.pid --cd /etc/'.
Program terminated with signal 6, Aborted.
#0  0x0000003362632925 in raise () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa8c98e47c0 (LWP 1422)):
#0  0x0000003362632925 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000003362634105 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fa8c87a198a in g_logv () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3  0x00007fa8c87a1a23 in g_log () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4  0x00007fa8c8797ba0 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#5  0x00007fa8c8797c53 in g_main_context_new () from /lib64/libglib-2.0.so.0
No symbol table info available.
#6  0x00007fa8c8ec9009 in pam_sm_authenticate () from /lib64/security/pam_fprintd.so
No symbol table info available.
#7  0x00007fa8c94d5cee in ?? () from /lib64/libpam.so.0
No symbol table info available.
#8  0x00007fa8c94d5600 in pam_authenticate () from /lib64/libpam.so.0
No symbol table info available.
#9  0x00007fa8c96e2c4d in openvpn_plugin_open_v1 () from /usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so

I have retained the rest of the abrt data, in case that is of further value.

rainbow6

Can you post your server config?

symphero

Sure thing:

port 1194
proto udp
dev tun
ca cacert.pem
cert myvpn.crt
key myvpn.key
dh dh2048.pem
server 192.168.1.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.0.0 255.255.255.0"
push "route 192.168.2.0 255.255.255.0"
push "route 192.168.3.0 255.255.255.0"
push "route 192.168.4.0 255.255.255.0"
push "redirect-gateway bypass-dhcp"
keepalive 10 120
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3
plugin /usr/lib64/openvpn/plugin/lib/openvpn-auth-pam.so login
crl-verify crl.pem

rainbow6

add / replace
status status.log 2
log openvpn.log
verb 3

is your cacert.pem generated by cacert or openssl self-signed?
also it is better to have the full path of your cert and keys in the config file.
ca /etc/openvpn/ca.crt
etc..

Check the openvpn.log once you start openvpn and see if there is any error/warning in there.

maybe its better to remove pam and test if the cert verification/authentication works first.

this is sample workable config you can based on;

#Enable if you are calling external scripts
#script-security 2

# Specify device
dev tun0
local x.x.x.x
port 443

server 172.16.0.0 255.255.255.0
topology subnet

# Certificates for VPN Authentication
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/dh2048.pem

#Certificate validation
crl-verify /etc/openvpn/crl.pem

# Routes/Gateway to push to the client
push "redirect-gateway def1"
push "dhcp-option DNS x.x.x.x"

# Make the link more resistent to connection failures keepalive
keepalive 20 60
persist-tun
persist-key
max-clients 60

float
mute-replay-warnings

# Run OpenVPN as a daemon and drop privileges to user/group nobody user nobody
user nobody
daemon

status status.log 10
status-version 2
log openvpn.log
verb 3

#Enable this if you want to have tls security checking
#tls-auth /etc/openvpn/keys/ta.key 0

#Performance tuning
txqueuelen 1000
sndbuf 131072
rcvbuf 131072

#SECURITY - use whichever suit you or just remove them to use default
cipher AES-128-CBC
# enable this if you using tls-auth
#tls-cipher TLS-DHE-RSA-WITH-AES-128-CBC-SHA