[SOLVED] Service crash at connexion since I enabled the revoke functionnality

thomas86

Hello,

I've been using OpenVPN with my dedicated server for a while. Everything was working fine until I decided I would revoke the certificate of one user.

I first I had some trouble to really understand all I needed to do to revoke users' certificate in a clean way, as I thought I only needed to delete the user's certificate files (which I didn't do). After searching on internet, I managed to understand how revocation works.

So after the revocation of my user,
I recreated the same user as the one I deleted (with same username), but since then, every time I try to connect to the VPN, the services crashes.

Everything was working perfectly fine until I did the revocation process.

What i did was :
- adding "crl-verify /etc/openvpn/easy-rsa/keys/crl.pem" command in "/etc/openvpn/server.conf"
- revoking the user using :
cd /etc/openvpn/easy-rsa/
./vars
./revoke-full <USERNAME>

As expected, those commands changed first letter of the line of the revoked user from V to R in the file keys/index.txt :

root@sd-00000:/etc/openvpn/easy-rsa# cat keys/index.txt
....
R       270113135359Z   170115142928Z   [b]04 [/b]     unknown /C=FR/ST=FR/L=Paris/O=server/OU=changeme/CN=<USERNAME>/name=changeme/emailAddress=admin@sd-00000.dedibox.fr

and added the revoked user's certificate to the crl.pem file :

root@sd-00000:/etc/openvpn/easy-rsa# openssl crl -in keys/crl.pem -text
Certificate Revocation List (CRL):
        Version 1 (0x0)
    Signature Algorithm: md5WithRSAEncryption
        Issuer: /C=FR/ST=FR/L=Paris/O=sd-30152/OU=changeme/CN=changeme/name=changeme/emailAddress=admin@sd-00000.dedibox.fr
        Last Update: Jan 15 14:29:28 2017 GMT
        Next Update: Feb 14 14:29:28 2017 GMT
Revoked Certificates:
    [b]Serial Number: 04[/b]
        Revocation Date: Jan 15 14:29:28 2017 GMT
    Signature Algorithm: md5WithRSAEncryption
         1e:0f:18:6f:13:11:b0:df:80:fe:7e:dd:ef:bb:0b:d2:6b:f4:
         c8:cf:4f:d9:cc:6e:33:60:a3:1b:b1:fa:f8:22:df:28:d0:b0:
         1c:f1:dc:be:58:4c:e4:95:6a:f2:35:87:3f:b2:71:f7:2f:eb:
         8e:9e:e1:af:70:d7:43:dc:7f:54:ac:4a:c1:f8:47:61:b2:6f:
         de:da:93:79:d2:95:62:76:e7:c9:d2:23:88:f4:4b:88:e9:e9:
         a7:3e:72:b8:82:d7:d1:9a:08:f2:23:29:41:f6:26:64:69:12:
         ec:32:ab:92:21:35:5f:ec:04:5e:b2:85:ad:61:45:68:0c:e7:
         b2:0d
-----BEGIN X509 CRL-----
MIIBfDCB5jANBgkqhkiG9w0BAQQFADCBoDELMAkGA1UEBhMCRlIxCzAJBgNVBAgT
AkZSMQ4wDAYDVQQHEwVQYXJpczERMA8GA1UEChMIc2QtMzAxNTIxETAPBgNVBAsT
CGNoYW5nZW1lMREwDwYDVQQDEwhjaGFuZ2VtZTERMA8GA1UEKRMIY2hhbmdlbWUx
KDAmBgkqhkiG9w0BCQEWGWFkbWluQHNkLTMwMTUyLmRlZGlib3guZnIXDTE3MDEx
NTE0MjkyOFoXDTE3MDIxNDE0MjkyOFowFDASAgEEFw0xNzAxMTUxNDI5MjhaMA0G
CSqGSIb3DQEBBAUAA4GBAB4PGG8TEbDfgP5+3e+7C9Jr9MjPT9nMbjNgoxux+vgi
3yjQsBzx3L5YTOSVavI1hz+ycfcv646e4a9w10Pcf1SsSsH4R2Gyb97ak3nSlWJ2
58nSI4j0S4jp6ac+criC19GaCPIjKUH2JmRpEuwyq5IhNV/sBF6yha1hRWgM57IN
-----END X509 CRL-----

Then I recreated the user, doing :

cd /etc/openvpn/easy-rsa
source vars
./build-key-pass <USERNAME>

The new user is visible in the keys/index.txt and the new certificates files have been issued.

root@sd-00000:/etc/openvpn/easy-rsa# cat keys/index.txt
....
V       270113144847Z           05      unknown /C=FR/ST=FR/L=Paris/O=server/OU=changeme/CN=<USERNAME>/name=changeme/emailAddress=admin@sd-00000.dedibox.fr

Then I copied the cert files to my client and tried to connect, but I couldn't.

So I tried to connect with another username (that was working before), and same, I couldn't connect anymore.

Config :

Server :
- Dedicated server hosted on internet at OneProvider.com, reachable directly with it's public IP address
- OS : Debian 7 up to date with openvpn (latest version)
- basic iptables rule to allow connections from and to the sslvpn port (I use the port TCP 110, same as POP, to make sure I can connect through my company's firewall)

Clients :
- My windows 7 PC with official OpenVPN client (connected from home, basic internet router)
- My android phone with official OpenVPN client (connected from eveywhere, wifi/3G/4G)
- On my PC I'm using cert auth only
- On my phone I'm using cert auth + password

Before trying to connect, the openvpn service is started and listening :

root@sd-00000:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN      3416/openvpn

When I start the connection on the client (Windows), I got those logs :

Sun Jan 15 17:59:55 2017 OpenVPN 2.4.0 x86_64-w64-mingw32 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Dec 27 2016
Sun Jan 15 17:59:55 2017 Windows version 6.1 (Windows 7) 64bit
Sun Jan 15 17:59:55 2017 library versions: OpenSSL 1.0.2i  22 Sep 2016, LZO 2.09
Enter Management Password:
Sun Jan 15 17:59:55 2017 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340
Sun Jan 15 17:59:55 2017 Need hold release from management interface, waiting...
Sun Jan 15 17:59:56 2017 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:25340
Sun Jan 15 17:59:56 2017 MANAGEMENT: CMD 'state on'
Sun Jan 15 17:59:56 2017 MANAGEMENT: CMD 'log all on'
Sun Jan 15 17:59:56 2017 MANAGEMENT: CMD 'hold off'
Sun Jan 15 17:59:56 2017 MANAGEMENT: CMD 'hold release'
Sun Jan 15 17:59:56 2017 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Sun Jan 15 17:59:56 2017 MANAGEMENT: >STATE:1484495996,RESOLVE,,,,,,
Sun Jan 15 17:59:56 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110
Sun Jan 15 17:59:56 2017 Socket Buffers: R=[8192->8192] S=[8192->8192]
Sun Jan 15 17:59:56 2017 Attempting to establish TCP connection with [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110 [nonblock]
Sun Jan 15 17:59:56 2017 MANAGEMENT: >STATE:1484495996,TCP_CONNECT,,,,,,
Sun Jan 15 17:59:57 2017 TCP connection established with [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110
Sun Jan 15 17:59:57 2017 TCP_CLIENT link local: (not bound)
Sun Jan 15 17:59:57 2017 TCP_CLIENT link remote: [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110
Sun Jan 15 17:59:57 2017 MANAGEMENT: >STATE:1484495997,WAIT,,,,,,
Sun Jan 15 17:59:57 2017 MANAGEMENT: >STATE:1484495997,AUTH,,,,,,
Sun Jan 15 17:59:57 2017 TLS: Initial packet from [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110, sid=072aed6c 4f45e09e
Sun Jan 15 17:59:58 2017 VERIFY OK: depth=1, C=FR, ST=FR, L=Paris, O=sd-00000, OU=changeme, CN=changeme, name=changeme, emailAddress=admin@sd-00000.dedibox.fr
Sun Jan 15 17:59:58 2017 VERIFY OK: depth=0, C=FR, ST=FR, L=Paris, O=sd-00000, OU=changeme, CN=changeme, name=changeme, emailAddress=admin@sd-00000.dedibox.fr
Sun Jan 15 17:59:58 2017 Connection reset, restarting [0]
Sun Jan 15 17:59:58 2017 SIGUSR1[soft,connection-reset] received, process restarting
Sun Jan 15 17:59:58 2017 MANAGEMENT: >STATE:1484495998,RECONNECTING,connection-reset,,,,,
Sun Jan 15 17:59:58 2017 Restart pause, 5 second(s)

And then, on the server the service is crashed :

root@sd-00000:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
.... (no openvpn line)

So when the clients try to auto-reconnect, the logs are the following :

Sun Jan 15 18:00:03 2017 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Sun Jan 15 18:00:03 2017 MANAGEMENT: >STATE:1484496003,RESOLVE,,,,,,
Sun Jan 15 18:00:03 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110
Sun Jan 15 18:00:03 2017 Socket Buffers: R=[8192->8192] S=[8192->8192]
Sun Jan 15 18:00:03 2017 Attempting to establish TCP connection with [AF_INET]<PUBLIC-IP-SERVER-ADDRESS>:110 [nonblock]
Sun Jan 15 18:00:03 2017 MANAGEMENT: >STATE:1484496003,TCP_CONNECT,,,,,,
Sun Jan 15 18:00:31 2017 SIGTERM[hard,init_instance] received, process exiting
Sun Jan 15 18:00:31 2017 MANAGEMENT: >STATE:1484496031,EXITING,init_instance,,,,,

During the crash, the server's logs are the following :

root@sd-00000:/etc/openvpn/easy-rsa# service openvpn start
[ ok ] Starting virtual private network daemon: server.

root@sd-00000:/etc/openvpn/easy-rsa# killall -USR2 openvpn ; tail -f /var/log/syslog
Jan 15 17:19:25 sd-00000 ovpn-server[3560]: MULTI: TCP INIT maxclients=1024 maxevents=1028
Jan 15 17:19:25 sd-00000 ovpn-server[3560]: Initialization Sequence Completed
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: OpenVPN CLIENT LIST
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: Updated,Sun Jan 15 17:19:28 2017
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: ROUTING TABLE
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: Virtual Address,Common Name,Real Address,Last Ref
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: GLOBAL STATS
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: Max bcast/mcast queue length,0
Jan 15 17:19:28 sd-00000 ovpn-server[3560]: END
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: MULTI: multi_create_instance called
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: Re-using SSL/TLS context
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: LZO compression initialized
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: Control Channel MTU parms [ L:1560 D:140 EF:40 EB:0 ET:0 EL:0 ]
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:135 ET:0 EL:0 AF:3/1 ]
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: Local Options hash (VER=V4): '79ef4284'
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: Expected Remote Options hash (VER=V4): '958c5492'
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: TCP connection established with [AF_INET]85.253.103.90:57830
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: TCPv4_SERVER link local: [undef]
Jan 15 17:19:37 sd-00000 ovpn-server[3560]: TCPv4_SERVER link remote: [AF_INET]85.253.103.90:57830
Jan 15 17:19:38 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 TLS: Initial packet from [AF_INET]85.253.103.90:57830, sid=d822a2c7 d5240121
[b]Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 CRL: cannot read: /etc/openvpn/easy-rsa/keys/crl.pem: Permission denied (errno=13)[/b]
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 Exiting
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 /sbin/route del -net 10.8.0.0 netmask 255.255.255.0
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 ERROR: Linux route delete command failed: external program exited with error status: 7
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 Closing TUN/TAP interface
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 /sbin/ifconfig tun0 0.0.0.0
Jan 15 17:19:39 sd-00000 ovpn-server[3560]: 85.253.103.90:57830 Linux ip addr del failed: external program exited with error status: 255

I saw that the service couldn't read "crl.pem", but when I check the rights, and only root had acccess to it, so I did a 755 chmod to allow the openvpn service to access this file

root@sd-00000:/etc/openvpn/easy-rsa# chmod 755 keys/crl.pem

I restarted the openvpn service, and tried to connect, and same problem (no connection + service crash), here are the logs :

root@sd-00000:/etc/openvpn/easy-rsa# killall -USR2 openvpn ; tail -f /var/log/syslog
Jan 15 17:41:56 sd-00000 ovpn-server[3876]: MULTI: TCP INIT maxclients=1024 maxevents=1028
Jan 15 17:41:56 sd-00000 ovpn-server[3876]: Initialization Sequence Completed
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: OpenVPN CLIENT LIST
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: Updated,Sun Jan 15 17:42:00 2017
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: ROUTING TABLE
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: Virtual Address,Common Name,Real Address,Last Ref
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: GLOBAL STATS
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: Max bcast/mcast queue length,0
Jan 15 17:42:00 sd-00000 ovpn-server[3876]: END
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: MULTI: multi_create_instance called
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: Re-using SSL/TLS context
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: LZO compression initialized
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: Control Channel MTU parms [ L:1560 D:140 EF:40 EB:0 ET:0 EL:0 ]
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:135 ET:0 EL:0 AF:3/1 ]
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: Local Options hash (VER=V4): '79ef4284'
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: Expected Remote Options hash (VER=V4): '958c5492'
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: TCP connection established with [AF_INET]85.253.103.90:58134
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: TCPv4_SERVER link local: [undef]
Jan 15 17:42:07 sd-00000 ovpn-server[3876]: TCPv4_SERVER link remote: [AF_INET]85.253.103.90:58134
Jan 15 17:42:08 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 TLS: Initial packet from [AF_INET]85.253.103.90:58134, sid=10199bd0 9e22396e
[b]Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 CRL: cannot read: /etc/openvpn/easy-rsa/keys/crl.pem: Permission denied (errno=13)[/b]
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 Exiting
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 /sbin/route del -net 10.8.0.0 netmask 255.255.255.0
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 ERROR: Linux route delete command failed: external program exited with error status: 7
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 Closing TUN/TAP interface
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 /sbin/ifconfig tun0 0.0.0.0
Jan 15 17:42:09 sd-00000 ovpn-server[3876]: 85.253.103.90:58134 Linux ip addr del failed: external program exited with error status: 255

And still the same error : Permission denied on crl.pem file.

Any ideas ?

Thanks

Thomas

TinCanTech

thomas86 wroteServer :
- Dedicated server hosted on internet at OneProvider.com, reachable directly with it's public IP address
- OS : Debian 7 up to date with openvpn (latest version)

Server config please ..

thomas86

I found where was the problem coming from, I'll still post the config below and explain what I did to fix the problem.

Server config :
root@dedi-fr-30152:/etc/openvpn# cat server.conf | egrep -v "(^#.*|^$|^;)"
port 110
proto tcp
dev tun
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
keepalive 10 20
cipher AES-256-CBC
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3
crl-verify /etc/openvpn/easy-rsa/keys/crl.pem

In the server config, I configured the user and group to run the openvpn service :
user nobody
group nogroup

When we use this setting in the same time with certificate revocation module, we have to make sure that the crl.pem file is readable by the user "nobody", so what I did is I gave "nobody" ownership of the "keys" folder :

chown -R nobody /etc/openvpn/easy-rsa/keys/

Problem solved, hope it helps 🙂

PS : how to mark the topic as solved ? 🙂

dazo

That is the crux of the issue. The CRL file is re-read on each connect, which requires the --user/--group user to be allowed to read the file and access the directory the CRL file is located in.

But ... seeing easy-rsa in the path ... you might be doing something wrong, it's off-topic from the initial issue. We don't have a good write-up of this issue yet, but here's one of my attempts on the mailing list: https://www.mail-archive.com/openvpn-users@lists.sourceforge.net/msg03466.html
TL;DR: Each time anyone installs a CA on an OpenVPN server accessible on the Internet, a kitten is hung up in a string, flogged alive and then slowly squeezed until it can't breathe. Please help us save these kittens!