I've done a fair bit of Googling to try and solve this issue and I made a bit of progress. At first, UFW definitely was blocking the client from responding to the server. This is no longer the case. The client is an Ubuntu 20.04 box and the server is running Debian 9. The server is a VPS in a datacenter somewhere and the client is on my home network.
I'm trying to do a pretty simple set up where just this one client connects to the server and the server will then run a reverse proxy to one of the client's services. Very, very briefly the set up worked after opening the firewall - I just restarted OpenVPN and I could wget the HTTP server running on the client. Then that didn't work, so I tried pinging - no luck there either.
The server is running UFW, the client is not running any firewall. I can see in the logs the TCP connection is working and a bit of data is transferred but I can't get anything beyond that working. I've reached the limit of my Google-fu too, 90% of the issues I've found that were similar were due to the firewall. This happens even if I disable UFW on the server and I've verified in the syslog nothing else is blocking it.
port 1194
# TCP or UDP server?
;proto tcp
proto tcp
dev tun
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth ta.key 0 # This file is secret
cipher AES-256-CBC
auth SHA256
max-clients 3
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log openvpn.log
verb 6
client
dev tun
proto tcp
remote <server name> 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
remote-cert-tls server
# I tried uncommenting this, no luck
;tls-auth ta.key 1
cipher AES-256-CBC
auth SHA256
verb 6
key-direction 1
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
#stripped certs out of the file, but they'd be here
openvpn-status.log
OpenVPN CLIENT LIST
Updated,Fri Oct 9 16:18:21 2020
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
machine,<client ip>:53960,13362,13975,Fri Oct 9 15:41:43 2020
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
10.8.0.6,machine,<client ip>:53960,Fri Oct 9 16:13:48 2020
GLOBAL STATS
Max bcast/mcast queue length,1
END
openvpn.log after client connected
Fri Oct 9 15:41:43 2020 us=9397 TCP connection established with [AF_INET]<client ip>:53960
Fri Oct 9 15:41:43 2020 us=9432 TCP_SERVER link local: (not bound)
Fri Oct 9 15:41:43 2020 us=9445 TCP_SERVER link remote: [AF_INET]<client ip>:53960
Fri Oct 9 15:41:43 2020 us=936713 <client ip>:53960 TCP_SERVER READ [54] from [AF_INET]<client ip>:53960: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 pid=[ #1 ] [ ] pid=0 DATA len=0
Fri Oct 9 15:41:43 2020 us=936772 <client ip>:53960 TLS: Initial packet from [AF_INET]<client ip>:53960, sid=ca951768 65f62c04
Fri Oct 9 15:41:43 2020 us=936830 <client ip>:53960 TCP_SERVER WRITE [66] to [AF_INET]<client ip>:53960: P_CONTROL_HARD_RESET_SERVER_V2 kid=0 pid=[ #1 ] [ 0 ] pid=0 DATA len=0
Fri Oct 9 15:41:44 2020 us=3564 <client ip>:53960 TCP_SERVER READ [62] from [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #2 ] [ 0 ]
Fri Oct 9 15:41:44 2020 us=125527 <client ip>:53960 TCP_SERVER READ [331] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #3 ] [ ] pid=1 DATA len=277
Fri Oct 9 15:41:44 2020 us=131439 <client ip>:53960 TCP_SERVER WRITE [1156] to [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #2 ] [ 1 ] pid=1 DATA len=1090
Fri Oct 9 15:41:44 2020 us=131544 <client ip>:53960 TCP_SERVER WRITE [1144] to [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #3 ] [ ] pid=2 DATA len=1090
Fri Oct 9 15:41:44 2020 us=131580 <client ip>:53960 TCP_SERVER WRITE [729] to [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #4 ] [ ] pid=3 DATA len=675
Fri Oct 9 15:41:44 2020 us=192555 <client ip>:53960 TCP_SERVER READ [62] from [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #4 ] [ 1 ]
Fri Oct 9 15:41:44 2020 us=253546 <client ip>:53960 TCP_SERVER READ [62] from [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #5 ] [ 2 ]
Fri Oct 9 15:41:44 2020 us=259173 <client ip>:53960 TCP_SERVER READ [1156] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #6 ] [ 3 ] pid=2 DATA len=1090
Fri Oct 9 15:41:44 2020 us=259270 <client ip>:53960 TCP_SERVER WRITE [62] to [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #5 ] [ 2 ]
Fri Oct 9 15:41:44 2020 us=259638 <client ip>:53960 TCP_SERVER READ [1144] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #7 ] [ ] pid=3 DATA len=1090
Fri Oct 9 15:41:44 2020 us=259693 <client ip>:53960 TCP_SERVER WRITE [62] to [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #6 ] [ 3 ]
Fri Oct 9 15:41:44 2020 us=319198 <client ip>:53960 TCP_SERVER READ [624] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #8 ] [ ] pid=4 DATA len=570
Fri Oct 9 15:41:44 2020 us=319527 <client ip>:53960 VERIFY OK: depth=1, CN=Miadola Software
Fri Oct 9 15:41:44 2020 us=319719 <client ip>:53960 VERIFY OK: depth=0, CN=machine
Fri Oct 9 15:41:44 2020 us=320214 <client ip>:53960 TCP_SERVER WRITE [117] to [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #7 ] [ 4 ] pid=4 DATA len=51
Fri Oct 9 15:41:44 2020 us=421288 <client ip>:53960 TCP_SERVER READ [487] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #9 ] [ 4 ] pid=5 DATA len=421
Fri Oct 9 15:41:44 2020 us=421392 <client ip>:53960 peer info: IV_VER=2.4.4
Fri Oct 9 15:41:44 2020 us=421432 <client ip>:53960 peer info: IV_PLAT=linux
Fri Oct 9 15:41:44 2020 us=421459 <client ip>:53960 peer info: IV_PROTO=2
Fri Oct 9 15:41:44 2020 us=421468 <client ip>:53960 peer info: IV_NCP=2
Fri Oct 9 15:41:44 2020 us=421476 <client ip>:53960 peer info: IV_LZ4=1
Fri Oct 9 15:41:44 2020 us=421483 <client ip>:53960 peer info: IV_LZ4v2=1
Fri Oct 9 15:41:44 2020 us=421491 <client ip>:53960 peer info: IV_LZO=1
Fri Oct 9 15:41:44 2020 us=421498 <client ip>:53960 peer info: IV_COMP_STUB=1
Fri Oct 9 15:41:44 2020 us=421506 <client ip>:53960 peer info: IV_COMP_STUBv2=1
Fri Oct 9 15:41:44 2020 us=421513 <client ip>:53960 peer info: IV_TCPNL=1
Fri Oct 9 15:41:44 2020 us=421587 <client ip>:53960 TCP_SERVER WRITE [319] to [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #8 ] [ 5 ] pid=5 DATA len=253
Fri Oct 9 15:41:44 2020 us=482402 <client ip>:53960 TCP_SERVER READ [62] from [AF_INET]<client ip>:53960: P_ACK_V1 kid=0 pid=[ #10 ] [ 5 ]
Fri Oct 9 15:41:44 2020 us=482484 <client ip>:53960 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 3072 bit RSA
Fri Oct 9 15:41:44 2020 us=482507 <client ip>:53960 [machine] Peer Connection Initiated with [AF_INET]<client ip>:53960
Fri Oct 9 15:41:44 2020 us=482538 machine/<client ip>:53960 MULTI_sva: pool returned IPv4=10.8.0.6, IPv6=(Not enabled)
Fri Oct 9 15:41:44 2020 us=482577 machine/<client ip>:53960 MULTI: Learn: 10.8.0.6 -> machine/<client ip>:53960
Fri Oct 9 15:41:44 2020 us=482587 machine/<client ip>:53960 MULTI: primary virtual IP for machine/<client ip>:53960: 10.8.0.6
Fri Oct 9 15:41:45 2020 us=744357 machine/<client ip>:53960 TCP_SERVER READ [96] from [AF_INET]<client ip>:53960: P_CONTROL_V1 kid=0 pid=[ #11 ] [ ] pid=6 DATA len=42
Fri Oct 9 15:41:45 2020 us=744461 machine/<client ip>:53960 PUSH: Received control message: 'PUSH_REQUEST'
Fri Oct 9 15:41:45 2020 us=744499 machine/<client ip>:53960 SENT CONTROL [machine]: 'PUSH_REPLY,route 10.8.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.8.0.6 10.8.0.5,peer-id 0,cipher AES-256-GCM' (status=1)
Fri Oct 9 15:41:45 2020 us=744519 machine/<client ip>:53960 Data Channel MTU parms [ L:1551 D:1450 EF:51 EB:406 ET:0 EL:3 ]
Fri Oct 9 15:41:45 2020 us=744596 machine/<client ip>:53960 Data Channel Encrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Oct 9 15:41:45 2020 us=744608 machine/<client ip>:53960 Data Channel Decrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Oct 9 16:13:48 2020 us=389531 machine/<client ip>:53960 TUN READ [60]
Fri Oct 9 16:17:52 2020 us=270001 machine/<client ip>:53960 TCP_SERVER READ [40] from [AF_INET]<client ip>:53960: P_DATA_V2 kid=0 DATA len=39
Fri Oct 9 16:18:00 2020 us=628687 machine/<client ip>:53960 TCP_SERVER WRITE [37] to [AF_INET]<client ip>:53960: P_DATA_V1 kid=0 DATA len=36
Fri Oct 9 16:18:02 2020 us=728477 machine/<client ip>:53960 TCP_SERVER READ [40] from [AF_INET]<client ip>:53960: P_DATA_V2 kid=0 DATA len=39
UFW rules
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), allow (routed)
New profiles: skip
To Action From
-- ------ ----
22 ALLOW IN Anywhere
80 ALLOW IN Anywhere
443 ALLOW IN Anywhere
27015 ALLOW IN Anywhere
27016 ALLOW IN Anywhere
27018 ALLOW IN Anywhere
25565 ALLOW IN Anywhere
1194/udp ALLOW IN Anywhere
22/tcp (OpenSSH) ALLOW IN Anywhere
Anywhere ALLOW IN 10.0.0.0/8
Anywhere on tun0 ALLOW IN Anywhere
1194/tcp ALLOW IN Anywhere
22 (v6) ALLOW IN Anywhere (v6)
80 (v6) ALLOW IN Anywhere (v6)
443 (v6) ALLOW IN Anywhere (v6)
27015 (v6) ALLOW IN Anywhere (v6)
27016 (v6) ALLOW IN Anywhere (v6)
27018 (v6) ALLOW IN Anywhere (v6)
25565 (v6) ALLOW IN Anywhere (v6)
1194/udp (v6) ALLOW IN Anywhere (v6)
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
Anywhere (v6) on tun0 ALLOW IN Anywhere (v6)
1194/tcp (v6) ALLOW IN Anywhere (v6)
10.0.0.0/8 ALLOW OUT Anywhere
Anywhere ALLOW OUT Anywhere on tun0
Anywhere (v6) ALLOW OUT Anywhere (v6) on tun0
