TLS Error: Handshake failed.Server is behind NAT on dyn ip

synthnassizer

Hi all,
this is a common problem that appears easily in searches , yet i can't seem to figure out what is wrong. Most results point to a problem with the firewall. I don't understand how to change

The problem appears when my openvpn server (which sits behind a NAT at my house) has its external IP changed, since my house internet connection has a dynamic ip.

SO, as soon as my external ip changes, my vpn clients can no longer connect to the server. It is a headache, because after a restart (of the openvpn server and perhaps the firewall as well) things usually are not corrected. After a few restarts though things will get working for a few days until my ip changes again.

My server config is the following:

cd /etc/openvpn
proto udp
port 32456
comp-lzo
verb 1
log-append /var/log/openvpn/server.log
status /var/log/openvpn/server-status.log
daemon
dev tun
persist-tun
persist-key
server 192.168.26.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
client-to-client
client-config-dir ccd
route-gateway 'dhcp'
route 192.168.114.0 255.255.255.0
route 192.168.18.0 255.255.255.0
push "route 192.168.112.0 255.255.255.0 vpn_gateway"
push "route 192.168.114.0 255.255.255.0"
push "route 192.168.18.0 255.255.255.0"
cipher BF-CBC
ca keys/ca.crt
dh keys/dh1024.pem
cert keys/stardust.crt
key keys/stardust.key
crl-verify certs/crl.pem
keepalive 20 120
user openvpn
group openvpn
max-clients 15

and the client config is the following (same across all clients)

client
dev tun
proto udp
remote my.dyndns1.site 32456
remote mydyndns2.site 32456
float
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/forge.crt
key /etc/openvpn/keys/forge.key
ns-cert-type server
cipher BF-CBC
comp-lzo
verb 3
mute 10
keepalive 10 60
log-append /var/log/openvpn/client.log

on the server , the log files states the following ()

Mon Jul 13 17:29:43 2015 us=15365 x.x.x.x:39397 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mon Jul 13 17:29:43 2015 us=15406 x.x.x.x:39397 TLS Error: TLS handshake failed
....
....(a few seconds pass by)
....
Mon Jul 13 17:29:43 2015 us=15492 x.x.x.x:39397 SIGUSR1[soft,tls-error] received, client-instance restarting
Mon Jul 13 17:29:44 2015 us=718669 MULTI: multi_create_instance called
Mon Jul 13 17:29:44 2015 us=718735 x.x.x.x:47092 Re-using SSL/TLS context
Mon Jul 13 17:29:44 2015 us=718770 x.x.x.x:47092 LZO compression initialized
Mon Jul 13 17:29:44 2015 us=718856 x.x.x.x:47092 Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Mon Jul 13 17:29:44 2015 us=718888 x.x.x.x:47092 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Mon Jul 13 17:29:44 2015 us=718932 x.x.x.x:47092 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SH
A1,keysize 128,key-method 2,tls-server'
Mon Jul 13 17:29:44 2015 us=718948 x.x.x.x:47092 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-C
BC,auth SHA1,keysize 128,key-method 2,tls-client'
Mon Jul 13 17:29:44 2015 us=718974 x.x.x.x:47092 Local Options hash (VER=V4): '530fdded'
Mon Jul 13 17:29:44 2015 us=718994 x.x.x.x:47092 Expected Remote Options hash (VER=V4): '41690919'
Mon Jul 13 17:29:44 2015 us=719028 x.x.x.x:47092 TLS: Initial packet from x.x.x.x:47092, sid=a439532b 854729a0

on the client , the log file states:

Mon Jul 13 17:32:51 2015 us=341010 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Mon Jul 13 17:32:51 2015 us=341263 Re-using SSL/TLS context
Mon Jul 13 17:32:51 2015 us=341309 LZO compression initialized
Mon Jul 13 17:32:51 2015 us=341451 Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Mon Jul 13 17:32:51 2015 us=341526 Socket Buffers: R=[229376->131072] S=[229376->131072]
Mon Jul 13 17:32:51 2015 us=510459 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Mon Jul 13 17:32:51 2015 us=510614 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128
,key-method 2,tls-client'
Mon Jul 13 17:32:51 2015 us=510642 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,k
eysize 128,key-method 2,tls-server'
Mon Jul 13 17:32:51 2015 us=510692 Local Options hash (VER=V4): '41690919'
Mon Jul 13 17:32:51 2015 us=510733 Expected Remote Options hash (VER=V4): '530fdded'
Mon Jul 13 17:32:51 2015 us=510771 UDPv4 link local: [undef]
Mon Jul 13 17:32:51 2015 us=510806 UDPv4 link remote: [AF_INET]x.x.x.x:32456
....
....(a few secs pass by)
....
Mon Jul 13 17:32:49 2015 us=340411 [UNDEF] Inactivity timeout (--ping-restart), restarting
Mon Jul 13 17:32:49 2015 us=340678 TCP/UDP: Closing socket
Mon Jul 13 17:32:49 2015 us=340752 SIGUSR1[soft,ping-restart] received, process restarting
Mon Jul 13 17:32:49 2015 us=340792 Restart pause, 2 second(s)

do you see anything wrong with my configs?
My guess is that the client firewalls do no allow ping-responses to reach them , once the ip of the server has changed (so all stateful ESTABLISH , RELATED connections no longer apply for the new server ip). Could this be the problem?
what do I do in this case?
Have you had this problem? How did you solve it?

thank you!

TiTex

does this file exists on your server crl-verify certs/crl.pem ?
comment out that option and test again

Traffic

synthnassizer wroteThe problem appears when my openvpn server (which sits behind a NAT at my house) has its external IP changed, since my house internet connection has a dynamic ip.

SO, as soon as my external ip changes, my vpn clients can no longer connect to the server. It is a headache, because after a restart (of the openvpn server and perhaps the firewall as well) things usually are not corrected. After a few restarts though things will get working for a few days until my ip changes again.

Have you setup a dynamic domain name ?

synthnassizer

@Titex the crl.pem file exists. Commenting out and restarting the server did not solve the problem
@Traffic. Yes of course I have a dyn dns service. In fact 2 of them from 2 different providers (no-ip and dtdns).
The client has 2 "remote" clauses and they are both FQDN from these 2 dyndns providers.

Traffic

synthnassizer wroteI have a dyn dns service. In fact 2 of them from 2 different providers

how often do they update ?

Also, do you have anything unusual in your client CCD file ?

synthnassizer

well one of the providers takes about 2-3 minutes to propagate the changes across DNS servers around the globe
the other one takes 30-40 minutes. I have notified them and they are looking into it..

in ccd I typically assign "static" ips to connecting clients. For example

ifconfig-push 192.168.26.42 192.168.26.41

on occasions I push the local DNS server as well

push "dhcp-option DNS 192.168.112.112"

That's about it.

Traffic

Sorry, we encountered an error while displaying this content. If you're a user, please try again later. If you're an administrator, take a look in your Flarum log files for more information.

synthnassizer

lol indeed,
I just swapped the messages to make some sequential sense in the snippet that I copied.
really the inactivity part comes after every waiting stage.. I just copied the previous inactivity msg and pasted it after the following link remote statement.
Rest assured I have not invented a time machine 😃

ok versions

server is openvpn-2.2.2-x86_64-1 (running on slackware 14.0)
clients vary:
openvpn-2.1.4-i486-2 (running on another slackware 13.37)
openvpn-2.2.2-x86_64-1 (running on another slackware 14.0)
2.2.1-8+deb7u3 (running on debian 7)
2.3.2-7ubuntu3.1 (running on xubuntu 14.04)
3.0.3core (?) (running on android phone)

To be honest now that you made me look into versions , I will say that the same version client, tends to connect when none of the others manage to connect. Hmmm I never heard/saw/thought that openvpn could be versions-incompatible. and perhaps it is not the openvpn itself but the ssl. could this be it?

Traffic

synthnassizer wroteHmmm I never heard/saw/thought that openvpn could be versions-incompatible.

OpenVPN is definitely sensitive to version incompatibilities .. but It will try to warn you in the log if serious incosistencies are detected. OpenVPN is in constant, active development and sometimes things go wrong. I highly recommend you try to get your versions inline and up-to-date.

synthnassizer wroteand perhaps it is not the openvpn itself but the ssl. could this be it?

OpenSSL versions could also theoretically cause problems .. but again, I think the errors would reflect if openssl is at fault.

Re-iterating:

Traffic wroteAlso, the problem you describe appears to be down to DNS update/resolving/cache ... this is fairly normal behavior and it comes down to how your DNS solution is setup.

synthnassizer

could you elaborate on the last bit?
I am not sure how the 2 minute DNS propagation would affect openvpn..

Traffic

synthnassizer wroteas soon as my external ip changes, my vpn clients can no longer connect to the server. It is a headache, because after a restart (of the openvpn server and perhaps the firewall as well) things usually are not corrected. After a few restarts though things will get working for a few days until my ip changes again.

Restarting your server has no effect on the problem .. you need to ensure your DDNS client updates your DDNS name.

synthnassizer

oh, well I am sure about that - have verified it actually.
Besides, this could not be the problem since there is an initial handshake between the client and the server (on the new ip address).

Traffic

synthnassizer wrotethere is an initial handshake between the client and the server (on the new ip address)

I'll have to take your word for it:

synthnassizer wroteMon Jul 13 17:32:51 2015 us=510806 UDPv4 link remote: [AF_INET]x.x.x.x:32456
....
....(a few secs pass by) *backwards* :mrgreen:
....
Mon Jul 13 17:32:49 2015 us=340411 [UNDEF] Inactivity timeout (--ping-restart), restarting
Try setting --verb 4 (server and client) and check the logs.

It is not required but --tls-auth is highly recommended as well :!:

synthnassizer

The logs above are taken with --verb 4.

I also tried getting the clients to level 2.2.2 and this did not solve the problem.

ok new info. found the problem, can't think of a solution:

I started checking at packets level and firewall. The problem in at the client's firewall. I realized that the replies of the server are not allowed to pass through the INPUT chain. Removing the firewall makes the client connect immediately.

The question is why does the client firewall not allow the server packets to get in. I tested with tcpdump command :

tcpdump -i eth1 host <server_ip> and udp

and got many of these:

17:31:22.467092 IP 192.168.233.165.36312 > <server_ip>.32456: UDP, length 14
17:31:22.510463 IP <server_ip>.13519 > 192.168.233.165.36312: UDP, length 26
17:31:24.553394 IP 192.168.233.165.36312 > <server_ip>.32456: UDP, length 14
17:31:24.595869 IP <server_ip>.13519 > 192.168.233.165.36312: UDP, length 26

so the client pings the server, then the server replies. which is good.
at the same time the client firewall seems to drop the server responses:

Jul 15 17:31:22 zeus INPUT packet died: IN=eth1 OUT= MAC=00:50:bf:2b:5f:f8:20:89:86:9a:f1:10:08:00 SRC=79.107.26.216 DST=192.168.233.165 LEN=54 TOS=00 PREC=0x00 TTL=56 ID=0 DF PROTO=UDP SPT=13519 DPT=36312 LEN=34
Jul 15 17:31:24 zeus INPUT packet died: IN=eth1 OUT= MAC=00:50:bf:2b:5f:f8:20:89:86:9a:f1:10:08:00 SRC=79.107.26.216 DST=192.168.233.165 LEN=54 TOS=00 PREC=0x00 TTL=56 ID=0 DF PROTO=UDP SPT=13519 DPT=36312 LEN=34

(see the timestamps match now 🙂

I have a stateful firewall, or at least I think I do :
- loading iptables does not output any error/warning messages
- lsmod shows:

ipt_ULOG 4757 14
iptable_mangle 1180 0
iptable_nat 3156 1
iptable_filter 1052 1
nf_conntrack_irc 2567 0
nf_nat_ftp 1272 0
nf_nat 11728 2 iptable_nat,nf_nat_ftp
nf_conntrack_ftp 4621 1 nf_nat_ftp
nf_conntrack_ipv4 9013 12 iptable_nat,nf_nat
nf_conntrack 42052 7 xt_state,iptable_nat,nf_conntrack_irc,nf_nat_ftp,nf_nat,nf_conntrack_ftp,nf_conntrack_ipv4
nf_defrag_ipv4 911 1 nf_conntrack_ipv4
ip_tables 8915 3 iptable_mangle,iptable_nat,iptable_filter
x_tables 10730 10 xt_limit,xt_pkttype,ipt_REJECT,xt_tcpudp,xt_state,ipt_ULOG,iptable_mangle,iptable_nat,iptable_filter,ip_tables

and finally, the firewall rules in input chain are:

echo "Process INPUT chain ..."
$IPT -A INPUT -p ALL -i $LO_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -j bad_packets
$IPT -A INPUT -p ALL -d 224.0.0.1 -j DROP
$IPT -A INPUT -p ALL -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A INPUT -p TCP -i $INET_IFACE -j tcp_inbound
$IPT -A INPUT -p UDP -i $INET_IFACE -j udp_inbound
$IPT -A INPUT -p ICMP -i $INET_IFACE -j icmp_packets
$IPT -A INPUT -m pkttype --pkt-type broadcast -j DROP
$IPT -A INPUT -m limit --limit 3/minute --limit-burst 3 -j ULOG --ulog-prefix "INPUT packet died: " --ulo
g-nlgroup 1

so of course, the packet originates from the $INET_IFACE. since it gets logged it means it goes through the whole chain. It should me accepted at the ESTABLISHED,RELATED line, but it is not. Why.... (sigh, oh why?).

Traffic

synthnassizer wroteThe logs above are taken with --verb 4.

Your configs do not reflect this ... :roll:

synthnassizer

That is true, but i did change the verb setting before getting the logs for this thread 🙂
Besides this level of log detail is only available in lvl 4. no more, no less.

also I was looking into conntrack.
The moment I see in the log

Wed Jul 15 18:42:53 2015 SIGUSR1[soft,tls-error] received, process restarting
Wed Jul 15 18:42:53 2015 Restart pause, 2 second(s)

tcpdump (with -vvvv appended) shows

18:40:50.167228 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 42)
192.168.233.165.49083 > <server_ip>.32456: [bad udp cksum ccfb!] UDP, length 14
18:40:50.212743 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 54)
<server_ip>.13519 > 192.168.233.165.49083: [udp sum ok] UDP, length 26

and

cat /proc/net/ip_conntrack | <server_ip>

udp 17 24 src=192.168.233.165 dst=<server_ip> sport=49083 dport=32456 [UNREPLIED] src=<server_ip> dst=192.168.233.165 sport=32456 dport=49083 mark=0 use=2

So the client sends a packet from port 49083 to the designated in the openvpn server config port 32456. Then the client firewall setups up a conntrack rule (for established connection). This is not established yet - i assume the server must reply on the same channel . BUT (reading back on the tcpdump, 2nd line) the openvpn server replies from port 13519, so this can never get passed the "ESTABLISHED,RELATED" lines. So the question is what can i do about this. Perhaps get the server to reply with sport=32456. Other alternatives?

Traffic

synthnassizer wroteBesides this level of log detail is only available in lvl 4. no more, no less

Actually it is --verb 4 (and higher) ..

synthnassizer wroteSo the client sends a packet from port 49083 to the designated in the openvpn server config port 32456. <snip> the openvpn server replies from port 13519

Probably your firewall.

synthnassizer wroteRemoving the firewall makes the client connect immediately.

Fairly conclusive proof you have a firewall problem.

synthnassizer

well there is 2 things to note here,
1) yes it is a firewall problem. I had made a notion of that in post #1. It is just that now we both have a very clear view of what happens.
2) I have been running openvpn servers on dynamic ip for the past 10 years almost. I have not always had this problem......

So while it is a firewall problem it seems something may be different on the openvpn side now. Arguably if openvpn responsed with sport the same as the listening port I would not have this problem. I understand this might not be possible - perhaps the kernel provides a network port at random to openvpn to respond and the latter does have a saying in this selection (i assume this Might be so - I have not read kernel documentation), but if it "used to work fine for many years" there probably is something that can be done (apart from paying for a static ip).

I would very much like to know the openvpn versions, configs, logs and of other users with servers running on dynamic ip and having both their server and clients behind NAT.....

Traffic

synthnassizer wroteArguably if openvpn responsed with sport the same as the listening port I would not have this problem.

OpenVPN does not respond using a different port .: the problem is somewhere else (firewall).

synthnassizer

OpenVPN does not respond using a different port .: the problem is somewhere else (firewall).

I don't understand. if openvpn does not respond from a difference source port then why does the reply packet leave from port 13519 and not 32456?

and what do you suggest I do about the firewall?