TLS Misconfiguration resulting in timeouts

Eniax

I feel like I'm close to getting this working, but the errors OpenVPN spits out are the same as if there is a network connectivity issue, so I can't diagnose. Something is wrong with my TLS configuration.

I've got an OpenVPN server configured, with 1194/udp opened on the firewall. I've confirmed I can ping this machine, and nmap the port. It's not blocked by an ISP either.

NOTE: I've also tried with tlsauth files and tls-crypt, but the result is the same.

Both server and client are runnin 2.4.6.
Client SSL: 1.1.0h-fips
Server SSL: 1.0.2k-fips

client

dev tun

proto udp

remote example.com 1194

;remote my-server-2 1194

resolv-retry infinite

nobind

user nobody

group nobody

persist-key

persist-tun

# SSL/TLS parms.

ca /home/user/Documents/vp-n/ca.crt

cert /home/user/Documents/vp-n/client.crt

key /home/ser/Documents/vp-n/client.key

remote-cert-tls server

tls-auth /home/user/Documents/vp-n/ta.key 1

cipher AES-256-CBC

remote-cert-eku "TLS Web Server Authentication"

tls-client

And my server config:

# Which TCP/UDP port should OpenVPN listen on?

port 1194

# TCP or UDP server?

proto udp

dev tun

ca /etc/openvpn/ssl/ca.crt

cert /etc/openvpn/ssl/server.crt

key /etc/openvpn/ssl/server.key # This file should be kept secret

# Diffie hellman parameters.

dh /etc/openvpn/ssl/dh.pem

# Network topology

topology subnet

# Configure server mode and supply a VPN subnet

server 10.8.0.0 255.255.255.0

ifconfig-pool-persist ipp.txt

push "redirect-gateway def1 bypass-dhcp"

push "dhcp-option DNS 198.206.14.241"

push "dhcp-option DNS 172.98.193.42"

keepalive 10 120

tls-auth /etc/openvpn/ssl/ta.key 0 # This file is secret

cipher AES-256-CBC

user nobody

group nobody

persist-key

persist-tun

status openvpn-status.log

explicit-exit-notify 1

##Remote TLS Authentication

remote-cert-eku "TLS Web Client Authentication"

Here is the log:

Thu Jul 26 10:38:29 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication

Thu Jul 26 10:38:29 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication

Thu Jul 26 10:38:29 2018 TCP/UDP: Preserving recently used remote address: [AF_INET]example.com:1194

Thu Jul 26 10:38:29 2018 Socket Buffers: R=[212992->212992] S=[212992->212992]

Thu Jul 26 10:38:29 2018 UDP link local (bound): [AF_INET][undef]:1194

Thu Jul 26 10:38:29 2018 UDP link remote: [AF_INET]example.com:1194

Thu Jul 26 10:38:29 2018 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay

Thu Jul 26 10:38:29 2018 TLS: Initial packet from [AF_INET]example.com:1194, sid=68679a68 09f015d0

Thu Jul 26 10:38:29 2018 VERIFY OK: depth=1, CN=example.com

Thu Jul 26 10:38:29 2018 VERIFY KU OK

Thu Jul 26 10:38:29 2018 Validating certificate extended key usage

Thu Jul 26 10:38:29 2018 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication

Thu Jul 26 10:38:29 2018 VERIFY EKU OK

Thu Jul 26 10:38:29 2018 VERIFY OK: depth=0, CN=example

Thu Jul 26 10:39:29 2018 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)

Thu Jul 26 10:39:29 2018 TLS Error: TLS handshake failed

Thu Jul 26 10:39:29 2018 SIGUSR1[soft,tls-error] received, process restarting

Thu Jul 26 10:39:29 2018 Restart pause, 5 second(s)

These lines repeat indefinitely.

Server Log:

Thu Jul 26 11:31:40 2018 TLS: Initial packet from [AF_INET], sid=d90e28d3 ba1b1be7

Thu Jul 26 11:31:40 2018 VERIFY ERROR: depth=0, error=unsupported certificate purpose: CN=client

Thu Jul 26 11:31:40 2018 OpenSSL: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed

Thu Jul 26 11:31:40 2018 TLS_ERROR: BIO read tls_read_plaintext error

Thu Jul 26 11:31:40 2018 TLS Error: TLS object -> incoming plaintext read error

Thu Jul 26 11:31:40 2018 TLS Error: TLS handshake failed

Thu Jul 26 11:31:40 2018 SIGUSR1[soft,tls-error] received, client-instance restarting

I feel like this is a misconfiguration of one of the config files. Any suggestions?

TinCanTech

Try removing these:

Eniax wrote

remote-cert-eku " .. "

What you are probably looking for is --remote-cert-tls

See --remote-cert-tls in The Manual v24x

See also the example files:
https://github.com/OpenVPN/openvpn/blob/57d6f103f378b927b9e0054b022b5b8b442973b8/sample/sample-config-files/client.conf#L104

Eniax

TinCanTech wrote Try removing these:
Eniax wrote
remote-cert-eku " .. "
What you are probably looking for is --remote-cert-tls

See --remote-cert-tls in The Manual v24x

See also the example files:
https://github.com/OpenVPN/openvpn/blob/57d6f103f378b927b9e0054b022b5b8b442973b8/sample/sample-config-files/client.conf#L104

Thank you. I cleaned up my config files and had a successful connection. Except now I cannot get out to the internet.

##############################################
# Sample client-side OpenVPN 2.0 config file #
# for connecting to multi-client server.     #
#                                            #
# This configuration can be used by multiple #
# clients, however each client should have   #
# its own cert and key files.                #
#                                            #
# On Windows, you might want to rename this  #
# file so it has a .ovpn extension           #
##############################################

# Specify that we are a client and that we
# will be pulling certain config file directives
# from the server.
client
tls-client
topology subnet
pull
# Use the same setting as you are using on
# the server.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
;dev tap
dev tun

proto udp

# The hostname/IP and port of the server.

remote my.public.ip  1194
;remote my-server-2 1194

resolv-retry infinite

# Most clients don't need to bind to
# a specific local port number.
nobind

# Downgrade privileges after initialization (non-Windows only)
;user nobody
;group nobody

# Try to preserve some state across restarts.
persist-key
persist-tun

# SSL/TLS parms.
# See the server config file for more
# description.  It's best to use
# a separate .crt/.key file pair
# for each client.  A single ca
# file can be used for all clients.
ca /etc/openvpn/ca.crt
cert /etc/openvpn/crouton.crt
key /etc/openvpn/easy-rsa/3/pki/private/crouton.key

remote-cert-tls server

# If a tls-auth key is used on the server
# then every client must also have the key.
tls-auth ta.key 1

cipher AES-256-CBC

# Enable compression on the VPN link.
# Don't enable this unless it is also
# enabled in the server config file.
#comp-lzo

# Set log file verbosity.
verb 3

# Silence repeating messages
;mute 20

Server conf:


# Which TCP/UDP port should OpenVPN listen on?
# If you want to run multiple OpenVPN instances
# on the same machine, use a different port
# number for each one.  You will need to
# open up this port on your firewall.
port 1194

# TCP or UDP server?
;proto tcp
proto udp

# "dev tun" will create a routed IP tunnel,
# "dev tap" will create an ethernet tunnel.
# Use "dev tap0" if you are ethernet bridging
# and have precreated a tap0 virtual interface
# and bridged it with your ethernet interface.
# If you want to control access policies
# over the VPN, you must create firewall
# rules for the the TUN/TAP interface.
# On non-Windows systems, you can give
# an explicit unit number, such as tun0.
# On Windows, use "dev-node" for this.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
;dev tap
dev tun

# Windows needs the TAP-Win32 adapter name
# from the Network Connections panel if you
# have more than one.  On XP SP2 or higher,
# you may need to selectively disable the
# Windows firewall for the TAP adapter.
# Non-Windows systems usually don't need this.
;dev-node MyTap

# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key).  Each client
# and the server must have their own cert and
# key file.  The server and all clients will
# use the same ca file.
#
# See the "easy-rsa" directory for a series
# of scripts for generating RSA certificates
# and private keys.  Remember to use
# a unique Common Name for the server
# and each of the client certificates.
#
# Any X509 key management system can be used.
# OpenVPN can also use a PKCS #12 formatted key file
# (see "pkcs12" directive in man page).
ca /etc/openvpn/easy-rsa/3/pki/ca.crt
cert /etc/openvpn/easy-rsa/3/pki/issued/server.crt
key /etc/openvpn/easy-rsa/3/pki/private/server.key  # This file should be kept secret

# Diffie hellman parameters.
# Generate your own with:
#   openssl dhparam -out dh2048.pem 2048
dh /etc/openvpn/easy-rsa/3/pki/dh.pem

# Network topology
# Should be subnet (addressing via IP)
# unless Windows clients v2.0.9 and lower have to
# be supported (then net30, i.e. a /30 per client)
# Defaults to net30 (not recommended)
topology subnet

# Configure server mode and supply a VPN subnet
# for OpenVPN to draw client addresses from.
# The server will take 10.8.0.1 for itself,
# the rest will be made available to clients.
# Each client will be able to reach the server
# on 10.8.0.1. Comment this line out if you are
# ethernet bridging. See the man page for more info.
server 10.8.0.0 255.255.255.0

# Maintain a record of client <-> virtual IP address
# associations in this file.  If OpenVPN goes down or
# is restarted, reconnecting clients can be assigned
# the same virtual IP address from the pool that was
# previously assigned.
ifconfig-pool-persist ipp.txt

# Push routes to the client to allow it
# to reach other private subnets behind
# the server.  Remember that these
# private subnets will also need
# to know to route the OpenVPN client
# address pool (10.8.0.0/255.255.255.0)
# back to the OpenVPN server.
;push "route 192.168.10.0 255.255.255.0"
;push "route 192.168.20.0 255.255.255.0"

# To assign specific IP addresses to specific
# clients or if a connecting client has a private
# subnet behind it that should also have VPN access,
# use the subdirectory "ccd" for client-specific
# configuration files (see man page for more info).

# EXAMPLE: Suppose the client
# having the certificate common name "Thelonious"
# also has a small subnet behind his connecting
# machine, such as 192.168.40.128/255.255.255.248.
# First, uncomment out these lines:
;client-config-dir ccd
;route 192.168.40.128 255.255.255.248
# Then create a file ccd/Thelonious with this line:
#   iroute 192.168.40.128 255.255.255.248
# This will allow Thelonious' private subnet to
# access the VPN.  This example will only work
# if you are routing, not bridging, i.e. you are
# using "dev tun" and "server" directives.

# EXAMPLE: Suppose you want to give
# Thelonious a fixed VPN IP address of 10.9.0.1.
# First uncomment out these lines:
;client-config-dir ccd
;route 10.9.0.0 255.255.255.252
# Then add this line to ccd/Thelonious:
#   ifconfig-push 10.9.0.1 10.9.0.2



# If enabled, this directive will configure
# all clients to redirect their default
# network gateway through the VPN, causing
# all IP traffic such as web browsing and
# and DNS lookups to go through the VPN
# (The OpenVPN server machine may need to NAT
# or bridge the TUN/TAP interface to the internet
# in order for this to work properly).
push "redirect-gateway def1 bypass-dhcp"

# Certain Windows-specific network settings
# can be pushed to clients, such as DNS
# or WINS server addresses.  CAVEAT:
# http://openvpn.net/faq.html#dhcpcaveats
# The addresses below refer to the public
# DNS servers provided by opendns.com.
push "dhcp-option DNS 10.8.0.1" #have also tried 8.8.8.8
push "dhcp-option DNS 8.8.4.4"

# The keepalive directive causes ping-like
# messages to be sent back and forth over
# the link so that each side knows when
# the other side has gone down.
# Ping every 10 seconds, assume that remote
# peer is down if no ping received during
# a 120 second time period.
keepalive 10 120

tls-auth /etc/openvpn/ssl/ta.key 0 # This file is secret

# Select a cryptographic cipher.
# This config item must be copied to
# the client config file as well.
# Note that v2.4 client/server will automatically
# negotiate AES-256-GCM in TLS mode.
# See also the ncp-cipher option in the manpage
cipher AES-256-CBC


# It's a good idea to reduce the OpenVPN
# daemon's privileges after initialization.
#
# You can uncomment this out on
# non-Windows systems.
user nobody
group nobody

# The persist options will try to avoid
# accessing certain resources on restart
# that may no longer be accessible because
# of the privilege downgrade.
persist-key
persist-tun

# Output a short status file showing
# current connections, truncated
# and rewritten every minute.
status openvpn-status.log

# By default, log messages will go to the syslog (or
# on Windows, if running as a service, they will go to
# the "\Program Files\OpenVPN\log" directory).
# Use log or log-append to override this default.
# "log" will truncate the log file on OpenVPN startup,
# while "log-append" will append to it.  Use one
# or the other (but not both).
log         openvpn.log
;log-append  openvpn.log

# Set the appropriate level of log
# file verbosity.
#
# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 3

# Silence repeating messages.  At most 20
# sequential messages of the same message
# category will be output to the log.
;mute 20

# Notify the client that when the server restarts so it
# can automatically reconnect.
explicit-exit-notify 1

Logs:
Server:

hu Jul 26 15:03:08 2018 library versions: OpenSSL 1.0.2k-fips  26 Jan 2017, LZO 2.06
Thu Jul 26 15:03:08 2018 Diffie-Hellman initialized with 2048 bit key
Thu Jul 26 15:03:08 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 26 15:03:08 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 26 15:03:08 2018 TUN/TAP device tun0 opened
Thu Jul 26 15:03:08 2018 TUN/TAP TX queue length set to 100
Thu Jul 26 15:03:08 2018 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Thu Jul 26 15:03:08 2018 /sbin/ip link set dev tun0 up mtu 1500
Thu Jul 26 15:03:08 2018 /sbin/ip addr add dev tun0 10.8.0.1/24 broadcast 10.8.0.255
Thu Jul 26 15:03:08 2018 Could not determine IPv4/IPv6 protocol. Using AF_INET
Thu Jul 26 15:03:08 2018 Socket Buffers: R=[212992->212992] S=[212992->212992]
Thu Jul 26 15:03:08 2018 UDPv4 link local (bound): [AF_INET][undef]:1194
Thu Jul 26 15:03:08 2018 UDPv4 link remote: [AF_UNSPEC]
Thu Jul 26 15:03:08 2018 GID set to nobody
Thu Jul 26 15:03:08 2018 UID set to nobody
Thu Jul 26 15:03:08 2018 MULTI: multi_init called, r=256 v=256
Thu Jul 26 15:03:08 2018 IFCONFIG POOL: base=10.8.0.2 size=252, ipv6=0
Thu Jul 26 15:03:08 2018 ifconfig_pool_read(), in='crouton,10.8.0.4', TODO: IPv6
Thu Jul 26 15:03:08 2018 succeeded -> ifconfig_pool_set()
Thu Jul 26 15:03:08 2018 IFCONFIG POOL LIST
Thu Jul 26 15:03:08 2018 crouton,10.8.0.4
Thu Jul 26 15:03:08 2018 Initialization Sequence Completed

Client:

Thu Jul 26 14:55:10 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 26 14:55:10 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 26 14:55:10 2018 TCP/UDP: Preserving recently used remote address: [AF_INET]xx.xx:1194
Thu Jul 26 14:55:10 2018 Socket Buffers: R=[212992->212992] S=[212992->212992]
Thu Jul 26 14:55:10 2018 UDP link local: (not bound)
Thu Jul 26 14:55:10 2018 UDP link remote: [AF_INET]xx.xx:1194
Thu Jul 26 14:55:11 2018 TLS: Initial packet from [AF_INET]xx.xx:1194, sid=dae146e7 2b50dcb7
Thu Jul 26 14:55:11 2018 VERIFY OK: depth=1, CN=server
Thu Jul 26 14:55:11 2018 VERIFY KU OK
Thu Jul 26 14:55:11 2018 Validating certificate extended key usage
Thu Jul 26 14:55:11 2018 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Thu Jul 26 14:55:11 2018 VERIFY EKU OK
Thu Jul 26 14:55:11 2018 VERIFY OK: depth=0, CN=server
Thu Jul 26 14:55:11 2018 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Thu Jul 26 14:55:11 2018 [server] Peer Connection Initiated with [AF_INET]xx.xxx.xx:1194
Thu Jul 26 14:55:12 2018 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Thu Jul 26 14:55:12 2018 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 10.8.0.1 ,dhcp-option DNS 8.8.4.4,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.8.0.4 255.255.255.0,peer-id 0,cipher AES-256-GCM'
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: timers and/or timeouts modified
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: --ifconfig/up options modified
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: route options modified
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: route-related options modified
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: peer-id set
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: adjusting link_mtu to 1624
Thu Jul 26 14:55:12 2018 OPTIONS IMPORT: data channel crypto options modified
Thu Jul 26 14:55:12 2018 Data Channel: using negotiated cipher 'AES-256-GCM'
Thu Jul 26 14:55:12 2018 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Thu Jul 26 14:55:12 2018 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Thu Jul 26 14:55:12 2018 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=wlp3s0 HWADDR=00:27:10:e1:a1:48
Thu Jul 26 14:55:12 2018 TUN/TAP device tun0 opened
Thu Jul 26 14:55:12 2018 TUN/TAP TX queue length set to 100
Thu Jul 26 14:55:12 2018 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Thu Jul 26 14:55:12 2018 /sbin/ip link set dev tun0 up mtu 1500
Thu Jul 26 14:55:12 2018 /sbin/ip addr add dev tun0 10.8.0.4/24 broadcast 10.8.0.255
Thu Jul 26 14:55:12 2018 /sbin/ip route add 162.220.51.35/32 via 192.168.0.1
Thu Jul 26 14:55:12 2018 /sbin/ip route add 0.0.0.0/1 via 10.8.0.1
Thu Jul 26 14:55:12 2018 /sbin/ip route add 128.0.0.0/1 via 10.8.0.1
Thu Jul 26 14:55:12 2018 Initialization Sequence Completed
^CThu Jul 26 14:56:01 2018 event_wait : Interrupted system call (code=4)
Thu Jul 26 14:56:01 2018 /sbin/ip route del 162.220.51.35/32
Thu Jul 26 14:56:01 2018 /sbin/ip route del 0.0.0.0/1
Thu Jul 26 14:56:01 2018 /sbin/ip route del 128.0.0.0/1
Thu Jul 26 14:56:01 2018 Closing TUN/TAP interface
Thu Jul 26 14:56:01 2018 /sbin/ip addr del dev tun0 10.8.0.4/24
Thu Jul 26 14:56:01 2018 SIGINT[hard,] received, process exiting

I can ping the new gateway (10.8.0.1) and i can ping the DNS, but no websites load and i cannot ping a website by its hostname

TinCanTech

Please see:
HOWTO: Routing all client traffic (including web-traffic) through the VPN

Eniax

TinCanTech wrote Please see:
HOWTO: Routing all client traffic (including web-traffic) through the VPN

This is the guide I followed. I use firewalld and added in

firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

but i also tried the iptables command as well. Both result in the same.
The system is configured to masquerade as well from firewall-cmd.

I really tried to do my due diligence prior to posting, but I feel like i've run out of options. Is there anything else I can try or some other way to test?

TinCanTech

I am guessing you have enabled IP_Forwarding ?
(To test: try ping the server eth0 IP from the client)

Anyway, because you appear to be running some flavour of *Nix,
you can use tcpdump to see the packet flow and work out what is going wrong.

Also, do not push a DNS server unless it is a DNS server .. and I doubt 10.8.0.1 is a DNS server.
Push google's DNS 8.8.8.8 . see if that helps.
Also, try pinging 8.8.8.8 from the client ..

Eniax

Yeah I'm on CentOS 7.
IP_Forwarding is configured in the /etc/systctl.conf file, and i've restarted the network.

The client is a Fedora 27 or 28 machine. I've used openvpn for so many years, but for some reason this setup (remote VPS) is just not working.

I updated the server.conf to have the google DNS as 1 and 2.
When connected, I can ping 8.8.8.8 and the gateway of 10.8.0.1, but I cannot ping any other website, nor look it up by hostname.

Here is the tcpdump for my google ping (and some random connections by firefox in the background)

tcpdump -D tun0:

16:34:30.262081 IP 10.8.0.4.33141 > cdns2.cox.net.domain: 50428+ A? detectportal.firefox.com.crouton(57)
16:34:33.600226 IP 10.8.0.4.34816 > cdns1.cox.net.domain: 19572+ A? google.com. (28)
16:34:33.604300 IP 10.8.0.4.34816 > cdns1.cox.net.domain: 54452+ AAAA? google.com. (28)
16:34:36.279991 IP 10.8.0.4.48427 > cdns1.cox.net.domain: 50428+ A? detectportal.firefox.com.crouton.oceana. (57)
16:34:41.274831 IP 10.8.0.4.40693 > cdns6.cox.net.domain: 50428+ A? detectportal.firefox.com.crouton.oceana. (57)
16:34:44.278181 IP 10.8.0.4.33141 > cdns2.cox.net.domain: 50428+ A? detectportal.firefox.com.crouton.oceana. (57)
16:34:50.287963 IP 10.8.0.4.47842 > cdns1.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:34:55.289987 IP 10.8.0.4.58198 > cdns6.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:34:58.294523 IP 10.8.0.4.50734 > cdns2.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:35:04.305978 IP 10.8.0.4.47842 > cdns1.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:35:09.306216 IP 10.8.0.4.58198 > cdns6.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:35:12.310164 IP 10.8.0.4.50734 > cdns2.cox.net.domain: 19095+ A? detectportal.firefox.com. (42)
16:35:18.313953 IP 10.8.0.4.37593 > cdns1.cox.net.domain: 7379+ A? detectportal.firefox.com.crouton(57)

I then ran
tcpdump -i eth0 udp port 1194 and then connected via the client.
I notice that traffic only goes one way, from the client to the server, when I make any request such as "ping google.com"

If I ping the gateway at 10.8.0.1, or the DNS at 8.8.8.8, I get back and forth responses.

Where could I check that my traffic was being filtered like this?
Is there a tcpdump command i can run to provide better information?

TinCanTech

Eniax wrote I then ran
tcpdump -i eth0 udp port 1194
<..>
Is there a tcpdump command i can run to provide better information?

The one you need is:

tcpdump -ni eth0 {with your preferred filter}

However,

Eniax wrote but for some reason this setup (remote VPS) is just not working

is probably the problem.

Try this instead of MASQ:

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 12.34.56.78   # <-- Use your OpenVPN server's real eth0 IP here

Eniax

TinCanTech wrote
The one you need is:

tcpdump -ni eth0 {with your preferred filter}

However,
is probably the problem.

Try this instead of MASQ:

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 12.34.56.78   # <-- Use your OpenVPN server's real eth0 IP here

I removed the masquerading directive from --direct --passthrough on firewalld and ran your command above.
I didn't see it in iptables -L, so I added it to firewalld like so:

firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 12.34.56.78]

with my real IP.

Still no success. So then I ran the ping command you had suggested:

07:38:06.173638 IP 12.34.56.78 > 5.235.119.24: ICMP host 12.34.56.78 unreachable - admin prohibited, length 68
07:38:07.203705 IP 12.34.56.78 > 5.235.119.24: ICMP host 12.34.56.78 unreachable - admin prohibited, length 68
07:38:09.342370 IP 12.34.56.78 > 77.72.85.25: ICMP host 12.34.56.78 unreachable - admin prohibited, length 48
07:38:11.144598 IP 12.34.56.78 > 5.126.230.49: ICMP host 12.34.56.78 unreachable - admin prohibited, length 68

I'm sorry for all the trouble, any time I've followed the HOWTO page for my home vpn, it's worked. This is the first time I've dealt with a VPS.

TinCanTech

Eniax wrote Still no success. So then I ran the ping command you had suggested:

07:38:06.173638 IP 12.34.56.78 > 5.235.119.24: ICMP host 12.34.56.78 unreachable - admin prohibited, length 68

I expect this is a firewall blocking you .. you could try -> 8.8.8.8

Eniax

8.8.8.8 works fine, as do any other DNS I tried (8.8.4.4 and all of OpenNIC's DNS machines), but no websites, by name or IP.

Not sure if I could test by putting the openvpn service in another zone?

TinCanTech

Eniax wrote 8.8.8.8 works fine

TinCanTech wrote do not push a DNS server unless it is a DNS server .. and I doubt 10.8.0.1 is a DNS server.
Push google's DNS 8.8.8.8 . see if that helps.

Sorry to repeat myself ..

Eniax wrote The client is a Fedora 27 or 28 machine

You probably need to import the pushed DNS.

Linux requires the following client directives to have DNS be successfully --push'ed

Client config:

script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

I am not sure about Fedora but I expect you will find that script present ..

Eniax

I apologize, I meant to mention that I updated the DNS in the server.conf after you'd mentioned to use a real DNS.

The update-resolv-conf doesn't seem to come pre-packaged any longer with the CentOS distribution of OpenVPN 2.4.6.

I did some searching and found the script on github, including one that is updated for systemd-resolved.
After adding it and updating the client.conf, this was the log output on the client:

Fri Jul 27 10:25:17 2018 /etc/openvpn/scripts/update-systemd-resolved tun0 1500 1552 10.8.0.4 255.255.255.0 init
<14>Jul 27 10:25:17 update-systemd-resolved: Link 'tun0' coming up
<14>Jul 27 10:25:17 update-systemd-resolved: Adding IPv4 DNS Server 8.8.8.8
<14>Jul 27 10:25:17 update-systemd-resolved: Adding IPv4 DNS Server 8.8.4.4
<14>Jul 27 10:25:17 update-systemd-resolved: SetLinkDNS(8 8 8 8 8 8 4 4)

which looks like it worked, but the outcome is still the same.

The firewall on the server side is what's getting me. In the past, simply adding masquerade and openvpn to the firewalld service has been sufficient. I've not even needed postrouting rules. I tried to stop firewalld altogether on the server, but that ended up blocking ALL traffic, so that wasn't it either.

TinCanTech

Eniax wrote Adding IPv4 DNS Server 8.8.8.8

Your client has at least picked up the DNS server that was pushed, good.

Eniax wrote The firewall on the server side is what's getting me. In the past, simply adding masquerade and openvpn to the firewalld service has been sufficient

It probably still is ..

Your tcpdump output seemed quite informative before .. try that again. I presume you know what it does ..

Eniax

TinCanTech wrote It probably still is ..

Your tcpdump output seemed quite informative before .. try that again. I presume you know what it does ..

You're absolutely correct.
Since updating the client.conf, etc, I can now curl any website by it's IP.
While connected to the VPN:

[user@client ~]$ curl 69.162.69.150
my.server.ip

69.162.69.150 is icanhazip.com

While connected normally:

[user@client ~]$ curl 69.162.69.150
my.normal.ip

What's interesting is I'm not seeing ANY icmp traffic from the client via tcpdump if I ping with a name. It seems that traffic isn't getting to the server at all.
I did some additional searching and found that if I manually set my /etc/resolv.conf to mirror the DNS server of the OpenVPN server, I can connect and resolve names. However, that resolution still does not go through the OpenVPN service, as shown by tcpdump.

DNS is something I've struggled to really understand in the past as well. Would this be a client or server misconfig?
The server can ping and curl websites no problem. IP or name.

TinCanTech

This is definitely the iptables rule you require:

TinCanTech wrote

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 12.34.56.78 # <-- Use your OpenVPN server's real eth0 IP here

because the server is a VPS and MASQ does not work.

I am quite sure this is the best way to configure your client to use the DNS pushed by the server:

Eniax wrote I did some searching and found the script on github, including one that is updated for systemd-resolved.
After adding it and updating the client.conf, this was the log output on the client:

Fri Jul 27 10:25:17 2018 /etc/openvpn/scripts/update-systemd-resolved tun0 1500 1552 10.8.0.4 255.255.255.0 init
<14>Jul 27 10:25:17 update-systemd-resolved: Link 'tun0' coming up
<14>Jul 27 10:25:17 update-systemd-resolved: Adding IPv4 DNS Server 8.8.8.8
<14>Jul 27 10:25:17 update-systemd-resolved: Adding IPv4 DNS Server 8.8.4.4
<14>Jul 27 10:25:17 update-systemd-resolved: SetLinkDNS(8 8 8 8 8 8 4 4)

Eniax wrote I did some additional searching and found that if I manually set my /etc/resolv.conf to mirror the DNS server of the OpenVPN server, I can connect and resolve names. However, that resolution still does not go through the OpenVPN service, as shown by tcpdump.

I do not know why your DNS resolution is not going via the tunnel .. tcpdump is your friend ..

Connect to your VPN and capture the packets on the server eth0 with tcpdump. Then from your cleint ping 8.8.8.8 and then try connecting to a website. If it does not work post the tcpdump output .. I'll take a look.

Eniax

TinCanTech wrote This is definitely the iptables rule you require:
TinCanTech wrote
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 12.34.56.78 # <-- Use your OpenVPN server's real eth0 IP here
because the server is a VPS and MASQ does not work.

I do not know why your DNS resolution is not going via the tunnel .. tcpdump is your friend ..

Connect to your VPN and capture the packets on the server eth0 with tcpdump. Then from your cleint ping 8.8.8.8 and then try connecting to a website. If it does not work post the tcpdump output .. I'll take a look.

I've updated firewalld to accommodate that rule.

While we can see that the resolved script is pushing the DNS, it doesn't seem to change the resolv.conf file at all.
I tried the older script that was available there and it didn't update resolv.conf either. Perhaps this has something to do with how systemd-resolved works?

Here is tcpdump, while the client is connected:

ping 8.8.8.8 <-- from client:

10:30:32.786735 IP 12.34.56.78 > 8.8.8.8: ICMP echo request, id 27784, seq 5, length 64
10:30:32.796919 IP 8.8.8.8 > 12.34.56.78: ICMP echo reply, id 27784, seq 5, length 64
10:30:33.788662 IP 12.34.56.78 > 8.8.8.8: ICMP echo request, id 27784, seq 6, length 64

So this is working. But, if I ping google.com (updating resolv.conf only has an effect on the client side, not the server side):

10:34:24.818949 IP 12.34.56.78 > 37.27.244.99: ICMP host 162.220.51.35 unreachable - admin prohibited, length 56
10:34:26.493026 IP 12.34.56.78 > 83.123.205.112: ICMP host 162.220.51.35 unreachable - admin prohibited, length 68
10:34:30.493813 IP 12.34.56.78 > 83.123.205.112: ICMP host 162.220.51.35 unreachable - admin prohibited, length 68
10:34:33.341308 IP 12.34.56.78 > 185.208.209.6: ICMP host 162.220.51.35 unreachable - admin prohibited, length 48

These lines happen regardless if the machine is being pinged, or a client is connected, so it's something unrelated. The notable thing is that pinging from the client while connected has no effect here.

Here is a traceroute from the client on 8.8.8.8:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  10.8.0.1 (10.8.0.1)  22.461 ms  22.419 ms  27.925 ms
 2  12.34.56.78 (12.34.56.78)  27.938 ms  27.931 ms  27.920 ms
.
.
.
 9  google-public-dns-a.google.com (8.8.8.8)  42.241 ms  42.123 ms  42.174 ms

the traceroute on google.com also passes this lineage as well.

Eniax

UPDATE:
It seems to be working...I think?

I went through and read a bit more about the update-systemd-resolved script. It seems that it requires and does not check if systemd-resolved is running, so will silently fail should it not update.

It did have me update nsswitch.conf and resolv.conf, which seem to be working properly now. Adding in 127.0.0.53 as my resolv.conf nameserver.
My main fear is that a NetworkManager restart will kill these settings.

However, here is the tcpdump from this new config:
Client:

ping icanhazip.com

PING icanhazip.com (69.162.69.150) 56(84) bytes of data.
64 bytes from icanhazip.com (69.162.69.150): icmp_seq=1 ttl=55 time=70.5 ms
64 bytes from icanhazip.com (69.162.69.150): icmp_seq=2 ttl=55 time=49.8 ms

Server:

tcpdump -ni eth0 icmp

11:08:09.734098 IP 12.34.56.78 > 69.162.69.150: ICMP echo request, id 29591, seq 5, length 64
11:08:09.755430 IP 69.162.69.150 > 12.34.56.78 : ICMP echo reply, id 29591, seq 5, length 64
11:08:10.732339 IP 12.34.56.78  > 69.162.69.150: ICMP echo request, id 29591, seq 6, length 64
11:08:10.753611 IP 69.162.69.150 > 12.34.56.78 : ICMP echo reply, id 29591, seq 6, length 64

Is there anything to look out for with NetworkManager or systemd-resolved? Will they clobber my config now if I restart the machine?

Thank you so much for all your help and patience. I know this was a rather unusual case.

TinCanTech

Eniax wrote It seems to be working...I think?

I think so too 😃

Eniax wrote Is there anything to look out for with NetworkManager or systemd-resolved? Will they clobber my config now if I restart the machine?

Simply do not use NetworkManager *until* you get the VPN working correctly without it.
Once the VPN works then you can integrate it into NM.

Arch Linux has a good start here:
https://wiki.archlinux.org/index.php/NetworkManager#Installation

systemd-resolved has not caused me any problems (yet) but I am no expert on that :oops:

Eniax wrote Thank you so much for all your help and patience. I know this was a rather unusual case

You have done an excellent job of clearly documenting and also researching your own problem
and it has been my pleasure to help you, so I thank you for an interesting and informative thread.

I don't say this often but ..
You may also like to subscribe to one/some of the many OpenVPN feeds. (mailing lists, IRC, github .. etc)

kind regards tct 8-)

Eniax

TinCanTech wrote
I don't say this often but ..
You may also like to subscribe to one/some of the many OpenVPN feeds. (mailing lists, IRC, github .. etc)

kind regards tct 8-)

:mrgreen: