Server fails to start after Ubuntu 16.04 LTS AWS update

Need help configuring your VPN? Just post here and you'll get that help.

Moderators: TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech

Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Post Reply
m9679
OpenVpn Newbie
Posts: 2
Joined: Sat Feb 10, 2018 9:40 pm

Server fails to start after Ubuntu 16.04 LTS AWS update

Post by m9679 » Sat Feb 10, 2018 10:03 pm

We fully updated our server recently and the openvpn service has failed to start since the reboot. It was running fine for month up until the updates/reboot, and there have been no changes to the openvpn service config.

Previous kernel version was: 4.4.0-1038-aws
New kernel version is: 4.4.0-1050-aws

openvpn package is the same version before an after: 2.4.4-xenial0

Server config inline below.

The only errors I can find are as follows:

Code: Select all

$systemctl status openvpn@tcp-443.service
● openvpn@tcp-443.service - OpenVPN connection to tcp-443
   Loaded: loaded (/lib/systemd/system/openvpn@.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/openvpn@.service.d
           └─override.conf
   Active: failed (Result: resources) since Sat 2018-02-10 21:48:54 UTC; 8s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
  Process: 2316 ExecStopPost=/usr/sbin/openvpn --rmtun --dev tun-%i (code=exited, status=0/SUCCESS)
  Process: 2313 ExecStartPost=/bin/sleep 0.1 (code=exited, status=0/SUCCESS)
  Process: 2285 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid (code=exited, status=0/SUCCESS)
  Process: 2280 ExecStartPre=/usr/sbin/openvpn --mktun --dev tun-%i --dev-type tun --user openvpn --group openvpn (code=exited, status=0/SUCCESS)
 Main PID: 1222 (code=exited, status=1/FAILURE)

Feb 10 21:48:54 redacted ovpn-tcp-443[2285]:   local = '[UNDEF]'
Feb 10 21:48:54 redacted ovpn-tcp-443[2285]:   local_port = '443'
Feb 10 21:48:54 redacted ovpn-tcp-443[2285]:   remote = '[UNDEF]'
Feb 10 21:48:54 redacted ovpn-tcp-443[2285]:   remote_port = '443'
Feb 10 21:48:54 redacted systemd[1]: openvpn@tcp-443.service: PID 2312 read from file /run/openvpn/tcp-443.pid does not exist or is a zombie.
Feb 10 21:48:54 redacted openvpn[2316]: Sat Feb 10 21:48:54 2018 TUN/TAP device tun-tcp-443 opened
Feb 10 21:48:54 redacted openvpn[2316]: Sat Feb 10 21:48:54 2018 Persist state set to: OFF
Feb 10 21:48:54 redacted systemd[1]: Failed to start OpenVPN connection to tcp-443.
Feb 10 21:48:54 redacted systemd[1]: openvpn@tcp-443.service: Unit entered failed state.
Feb 10 21:48:54 redacted systemd[1]: openvpn@tcp-443.service: Failed with result 'resources'.

Code: Select all

$journalctl -xe
<snip/>
Linux ip link set failed: external program exited with error status: 2
Feb 10 21:48:54 redacted ovpn-tcp-443[2312]: Exiting due to fatal error
Feb 10 21:48:54 redacted systemd[1]: openvpn@tcp-443.service: PID 2312 read from file /run/openvpn/tcp-443.pid does not exist or is a zombie.
Feb 10 21:48:54 redacted openvpn[2316]: Sat Feb 10 21:48:54 2018 TUN/TAP device tun-tcp-443 opened
Feb 10 21:48:54 redacted openvpn[2316]: Sat Feb 10 21:48:54 2018 Persist state set to: OFF
Feb 10 21:48:54 redacted audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=openvpn@tcp-443 comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Feb 10 21:48:54 redacted audispd[721]: node=redacted type=SERVICE_START msg=audit(1518299334.546:276): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=openvpn@tcp-443 comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Feb 10 21:48:54 redacted systemd[1]: Failed to start OpenVPN connection to tcp-443.
-- Subject: Unit openvpn@tcp-443.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit openvpn@tcp-443.service has failed.
--
-- The result is failed.
There is nothing else in /var/log/syslog around this error (just the same messages as above).

I've seen posts relating to failures to install the ip routes, but that is a different error message from "Linux ip link set failed", and it makes no difference if I remove the

Code: Select all

push "route 10.0.99.0   255.255.255.192"
lines.

Can't see anyone else reporting this. The smoking gun is that it worked before but not after the updates/reboot.
I have rebooted a few times since then for good measure, but no dice.
This is running on a t2.micro and has ample resources for starting the service (I think the reference to 'resources' in the error trace just means it can't create the tunnel interface?)


Anything else I can try/check?



[oconf=]
dev tun-tcp-443
server 10.237.8.0 255.255.255.0
topology subnet
push "route 10.0.99.0 255.255.255.192" # Public 1
push "route 10.0.99.64 255.255.255.192" # Public 2
push "route 10.0.201.0 255.255.255.192" # Private 1A
push "route 10.0.201.64 255.255.255.192" # Private 2A
push "route 10.0.202.0 255.255.255.192" # Private 1B
push "route 10.0.202.64 255.255.255.192" # Private 2B
ifconfig-pool-persist ipp.txt
#push "dhcp-option DNS 10.237.8.1"
proto tcp
port 443

ca /etc/openvpn/pki/ca.crt
cert /etc/openvpn/pki/issued/server@redeacted.crt
key /etc/openvpn/pki/private/server@redacted.key
dh /etc/openvpn/pki/dh.pem
crl-verify /etc/openvpn/pki/crl.pem

# Fix for the Windows 10 DNS leak described here:
# https://community.openvpn.net/openvpn/ticket/605
push block-outside-dns

remote-cert-tls client
keepalive 10 120
tls-auth /etc/openvpn/pki/ta.key 0
cipher AES-256-CBC
tls-cipher TLS-DHE-DSS-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-DSS-WITH-AES-256-CBC-SHA256
auth SHA512
tls-server
tls-version-min 1.2
comp-lzo
persist-key
persist-tun
#status openvpn-status.log
#log-append openvpn.log
verb 11
[/oconf]

m9679
OpenVpn Newbie
Posts: 2
Joined: Sat Feb 10, 2018 9:40 pm

Re: Server fails to start after Ubuntu 16.04 LTS AWS update

Post by m9679 » Sun Mar 18, 2018 4:49 pm

In case this helps others, I'm pretty sure this was caused by one of the following settings being reverted following apt upgrades:
- cap_net_admin+eip
- cap_net_bind_service+eip
- cap_net_admin+eip

We use an orchestration tool and the solution was to re-run the orchestration after the apt upgrades (and each time after future upgrades, probably).

TiTex
OpenVPN Super User
Posts: 310
Joined: Tue Apr 12, 2011 6:22 am

Re: Server fails to start after Ubuntu 16.04 LTS AWS update

Post by TiTex » Sun Mar 18, 2018 7:54 pm

where would those options be set ? ... i'm guessing it's something aws specific since there an "+eip" in each of them (or it does not mean elastic ip ?)
also systemd keeps messing things up trying to do everything lately on a linux box , like a giant bloatware that it is (conjobs,mounts,service management,network stuff, date & time settings) it's like an octopus ... all over the place :)

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: Server fails to start after Ubuntu 16.04 LTS AWS update

Post by TinCanTech » Sun Mar 18, 2018 8:28 pm

m9679 wrote:
Sat Feb 10, 2018 10:03 pm
Anything else I can try/check?
Your openvpn log.

See --log & --verb in The Manual v24x

FYI: verb 11 is way too much .. verb 4 is sufficient.

TiTex
OpenVPN Super User
Posts: 310
Joined: Tue Apr 12, 2011 6:22 am

Re: Server fails to start after Ubuntu 16.04 LTS AWS update

Post by TiTex » Sun Mar 18, 2018 8:42 pm

@TinCanTech , openvpn logs don't really help when the linux capabilities ( http://man7.org/linux/man-pages/man7/ca ... ies.7.html ) are in play
as far as i know , you don't get much info except for the generic "failed" message.
i'm guessing it's something aws specific since there an "+eip" in each of them (or it does not mean elastic ip ?)
nevermind , found the answer in https://mirrors.edge.kernel.org/pub/lin ... aq-0.2.txt
Last edited by TiTex on Sun Mar 18, 2018 8:53 pm, edited 1 time in total.

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: Server fails to start after Ubuntu 16.04 LTS AWS update

Post by TinCanTech » Sun Mar 18, 2018 8:52 pm

@Titex, understood .. but the openvpn server log usually has more info than the systemd output.

Post Reply