Synology Server: TLS key negociation failed to occur

Need help configuring your VPN? Just post here and you'll get that help.

Moderators: TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech

Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Post Reply
synology_rs816
OpenVpn Newbie
Posts: 2
Joined: Mon Dec 06, 2021 10:17 am

Synology Server: TLS key negociation failed to occur

Post by synology_rs816 » Mon Dec 06, 2021 10:30 am

Dear All,
first of all I would like to anticipate that I saw another similar topic, and I can confirm you that that didn't help me.
I'm the owner of a Synology RS816 and I install the OOTB package for the VPN Server.
As you might imagine, I configured OpenVPN and it worked really well on the DSM v6 of Synology, without giving any errors.

Recently I had the good idea to upgrade the latest release v7, but all at a sudden the VPN Server stopped working giving these messages:

TLS Error: TLS key negociation failed to occur within 60 seconds (check your network connectivity)
TLS Error: TLS handshake failed

Meanwhile Tunnelblick remains stuck in the status "Waiting for server response.."

1) I accessed the Synology Control Panel and I opened Security and I create a new LetsEncrypt certificate and it was successful (all marked in green).
2) still in the certificate settings, I made sure that the VPN Server is using my new generated certificate
3) I opened the firewall section, I enabled the firewall and it's notificaiton and i created a custom rule with Allow All/Allow All/Allow All (not best practice, but it's only for debug reasons)
4) I access the VPN Server and in General Settings I make sure that the correct Network Interface is selected
5) In privileges I make sure that my user has the OpenVPN [v} check
6) in the OpenVPN settings I make sure that there is the port 1194, with:
- Enable compression on the VPN link
- Allow clients to access server's LAN
- Verify TLS auth key

I exported the configuration, I get the ovpn file, I modify the DNS

I make sure that my router has the port forwarding on 1194 (like it was before).

I save, I import the certificate on TunnelBlick, but I get again these errors:

TLS Error: TLS key negociation failed to occur within 60 seconds (check your network connectivity)
TLS Error: TLS handshake failed

Might you have any ideas? I would like to avoid to reset the NAS to get back to the DSM v6, thanks!

synology_rs816
OpenVpn Newbie
Posts: 2
Joined: Mon Dec 06, 2021 10:17 am

Re: Synology Server: TLS key negociation failed to occur

Post by synology_rs816 » Thu Dec 09, 2021 9:02 am

We found the problem, apparently in the latest release of OpenVPN on Synology, there is an issue when using the UDP protocol.
We had to use the TCP and apply some different modification on the ovpn file extracted, like forcing the TLS, and some other stuffs, that in your blog are really well described.
Until no new release of the VPN Server will be released, we will keep stick to TCP.
Hope that helped someone.

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1333
Joined: Tue Feb 16, 2021 10:41 am

Re: Synology Server: TLS key negociation failed to occur

Post by openvpn_inc » Sat Dec 11, 2021 4:58 pm

Hi,

It won't help. It's bad advice. Better advice is to fix whatever they broke in UDP. Or, run the server on a better platform.

Regards, rob0
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

Post Reply