Hi all, I've recently been trying to find out why some users of our openvpn setup are having regular drop outs. We run the server on ubuntu trusty, and were using the default package in their repos (2.3 I think). Our server is on an Gigabit Virgin Media Fibre connection. At this stage, my own connection was dropping out every 1-2 minutes when I used it, I have 100Mbps Virgin Media Fibre at home. I'm using Fedora, and currently have openvpn-2.4.5-1.fc27.x86_64 installed. I made the decision to upgrade the server side software to the build.openvpn.net repo which gave me 2.4.4-trusty0 This solved my problems with VPN and mine is now rock solid, and only drops out very occasionally, even on a 4g hotspot. However, at this point, some of our users started reporting issues where their VPN was dropping out periodically. Some of these users were on openSUSE, ubuntu 16.04 and OSX using Tunnelblick latest stable. A sample of our VPN server config is: [oconf=Server Config]port 1194 proto udp dev tun ca ca.crt cert vpn.crt key vpn.key dh dh2048.pem server 10.10.4.0 255.255.255.0 ifconfig-pool-persist ipp.txt push "route 10.10.8.0 255.255.248.0" push "redirect-gateway def1 bypass-dhcp" push "dhcp-option DNS 10.10.10.3" push "dhcp-option DNS 10.10.10.6" push "dhcp-option DNS 8.8.8.8" push "dhcp-option DOMAIN domain.com" keepalive 10 120 comp-lzo user nobody group nogroup persist-key persist-tun status openvpn-status.log log-append /var/log/openvpn.log verb 3 reneg-sec 0 [/oconf] Also, our client config: [oconf=Client Config]remote vpn.domain.com proto udp port 1194 comp-lzo verb 3 dev tun persist-tun persist-key client cipher BF-CBC ca ca.crt cert user.crt key user.key auth-user-pass reneg-sec 0[/oconf] Around the time of the user dropping out I would see messages like the following: [code]TLS Error: local/remote TLS keys are out of sync[/code] These lines would repeat, sometimes a user would drop connection, other times it would be fine. I found an article from some years ago saying that this is common with UDP openvpn as the client can't be notified when the server has restarted. I don't believe the server had been restarted, but as a test I created a TCP VPN, which was mostly the same, but had no comp-lzo enabled. Again for my connection this worked fine, but some users still saw sporadic drops. One user reported that if he were downloading any large file while on the VPN it made it more unstable. We also looked in to whether this was caused by MTU settings, the user who saw some major issues found an article that suggested finding the highest value of MTU that worked by doing a ping check and using that value as the mssfix in the config. We tried this, and when he tested he found he couldn't get ping to work until he set his MTU around 500, at this value the VPN wouldn't even connect. I know this user has a slightly poor internet connection (they are on ADSL on fairly poor lines) but I would hope the VPN to be stable even on fairly poor internet connections. At this point I am running out of things to try, but I have some questions: [list=square] [*]Should different versions of openvpn client/server cause stability issues? Since some users use tunnelblick keeping the versions matching is quite difficult. [*]Is there anything else I can try to help our users gain a solid VPN connection using openvpn? I have now been asked to start looking for alternatives to openvpn, as the stability issues is making it unusable for some of our users. But I would rather not replace it if it can be fixed.[/list] I am aware that comp-lzo is deprecated in 2.5 and we need to move away from it. But I don't see any point fixing that unless I know I am going to continue to support openvpn at this point.

[quote=Rumbles post_id=77820 time=1522226675 user_id=43707] TLS Error: local/remote TLS keys are out of sync [/quote]I have also experienced this .. It is very difficult to explain because I do not have a complete answer. The first thing I notice is you are using --[b]reneg-sec 0[/b] .. this is a very bad decision. The default value is 3600 for a reason, that is after such a period the data channel can be exposed to [url=https://community.openvpn.net/openvpn/wiki/SWEET32]SWEET32[/url] attack (Depending on amount of data transferred). You can and should mitigate this by using --[b]cipher AES-256-CBC[/b] However, Openvpn 2.4.x now comes with [i]Negotiable Cipher Protocol[/i] (NCP) which will negotiate to cipher AES-256-GCM for clients which support it (You mentioned you have lots of different clients, some of them may not be able to use this). And this is where we start [i]down the rabbit hole[/i] .. NCP is very new and is known to have some configuration issues .. however, fixes for these issues will take time because they are not fully understood as yet. What you can do is .. [list]1. Disable NCP with --[b]ncp-disable[/b] 2. Specify --[b]cipher AES-256-CBC[/b] in your server and [b]all[/b] clients[/list] I would also recommend you contact the developers at the users-mailing list for more information.

[quote=Rumbles post_id=77833 time=1522247152 user_id=43707] making those changes, but will they actually improve the number of drops we are seeing? [/quote]I believe by disabling NCP the particular error you are seeing should go away completely (at least for 99% of connection) but that is only my personal experience not an official answer .. [quote=Rumbles post_id=77833 time=1522247152 user_id=43707] Are you saying NCP is a likely cause for the drops and disabling it might help? [/quote]I believe so .. [quote=Rumbles post_id=77833 time=1522247152 user_id=43707] But if I do that I should change ciphers at the same time [/quote]Yes .. because your current cipher is blowfish (the default) [quote=Rumbles post_id=77833 time=1522247152 user_id=43707] If I change to AES-256-CBC do I need to change the reneg-sec setting also? [/quote]I would recommend not using --reneg 0 ever .. unless you implement one of the other reneg parameters. [quote=Rumbles post_id=77833 time=1522247152 user_id=43707] Is there any way to check whether NCP has kicked in [/quote]It is on by default (ovpn 2.4.x) , disable with --ncp-disable .. and check your logs.

[quote=TinCanTech post_id=77835 time=1522247654 user_id=37053] [quote=Rumbles post_id=77833 time=1522247152 user_id=43707] If I change to AES-256-CBC do I need to change the reneg-sec setting also? [/quote]I would recommend not using --reneg 0 ever .. unless you implement one of the other reneg parameters. [/quote] Thanks again, can you clarify what other reneg parameters you can use? I have been told by a colleague that they need reneg set to 0 (as we use the Duo plugin for 2FA and they recommend it), so I could do with having a discussion with them about it, but would want to know what parameters you are referring to.

OpenVPN drops out regularly for some users

Rumbles

Hi all,

I've recently been trying to find out why some users of our openvpn setup are having regular drop outs.

We run the server on ubuntu trusty, and were using the default package in their repos (2.3 I think). Our server is on an Gigabit Virgin Media Fibre connection.

At this stage, my own connection was dropping out every 1-2 minutes when I used it, I have 100Mbps Virgin Media Fibre at home. I'm using Fedora, and currently have openvpn-2.4.5-1.fc27.x86_64 installed.

I made the decision to upgrade the server side software to the build.openvpn.net repo which gave me 2.4.4-trusty0

This solved my problems with VPN and mine is now rock solid, and only drops out very occasionally, even on a 4g hotspot.

However, at this point, some of our users started reporting issues where their VPN was dropping out periodically. Some of these users were on openSUSE, ubuntu 16.04 and OSX using Tunnelblick latest stable.

A sample of our VPN server config is:

port 1194
proto udp
dev tun
ca ca.crt
cert vpn.crt
key vpn.key
dh dh2048.pem
server 10.10.4.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 10.10.8.0 255.255.248.0"
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 10.10.10.3"
push "dhcp-option DNS 10.10.10.6"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DOMAIN domain.com"
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
reneg-sec 0

Also, our client config:

remote vpn.domain.com
proto udp
port 1194
comp-lzo
verb 3
dev tun
persist-tun
persist-key
client
cipher BF-CBC
ca ca.crt
cert user.crt
key user.key
auth-user-pass
reneg-sec 0

Around the time of the user dropping out I would see messages like the following:

TLS Error: local/remote TLS keys are out of sync

These lines would repeat, sometimes a user would drop connection, other times it would be fine. I found an article from some years ago saying that this is common with UDP openvpn as the client can't be notified when the server has restarted. I don't believe the server had been restarted, but as a test I created a TCP VPN, which was mostly the same, but had no comp-lzo enabled. Again for my connection this worked fine, but some users still saw sporadic drops. One user reported that if he were downloading any large file while on the VPN it made it more unstable.

We also looked in to whether this was caused by MTU settings, the user who saw some major issues found an article that suggested finding the highest value of MTU that worked by doing a ping check and using that value as the mssfix in the config. We tried this, and when he tested he found he couldn't get ping to work until he set his MTU around 500, at this value the VPN wouldn't even connect. I know this user has a slightly poor internet connection (they are on ADSL on fairly poor lines) but I would hope the VPN to be stable even on fairly poor internet connections.

At this point I am running out of things to try, but I have some questions:

Should different versions of openvpn client/server cause stability issues? Since some users use tunnelblick keeping the versions matching is quite difficult.
Is there anything else I can try to help our users gain a solid VPN connection using openvpn? I have now been asked to start looking for alternatives to openvpn, as the stability issues is making it unusable for some of our users. But I would rather not replace it if it can be fixed.

I am aware that comp-lzo is deprecated in 2.5 and we need to move away from it. But I don't see any point fixing that unless I know I am going to continue to support openvpn at this point.

TinCanTech

Rumbles wrote TLS Error: local/remote TLS keys are out of sync

I have also experienced this ..

It is very difficult to explain because I do not have a complete answer.

The first thing I notice is you are using --reneg-sec 0 .. this is a very bad decision.
The default value is 3600 for a reason, that is after such a period the data channel can be exposed to SWEET32 attack (Depending on amount of data transferred).
You can and should mitigate this by using --cipher AES-256-CBC

However, Openvpn 2.4.x now comes with Negotiable Cipher Protocol (NCP) which will negotiate to cipher AES-256-GCM for clients which support it (You mentioned you have lots of different clients, some of them may not be able to use this).

And this is where we start down the rabbit hole ..

NCP is very new and is known to have some configuration issues .. however, fixes for these issues will take time because they are not fully understood as yet.

What you can do is ..

1. Disable NCP with --ncp-disable
2. Specify --cipher AES-256-CBC in your server and all clients

I would also recommend you contact the developers at the users-mailing list for more information.

Rumbles

Thanks for that info, I can definitely look in to making those changes, but will they actually improve the number of drops we are seeing? Reading this I get the impression these changes would affect security more than anything.

Are you saying NCP is a likely cause for the drops and disabling it might help? But if I do that I should change ciphers at the same time. If I change to AES-256-CBC do I need to change the reneg-sec setting also?

Is there any way to check whether NCP has kicked in and changed the cipher a connection is using?

EDIT:

Sorry, I see it in the logs:

Wed Mar 28 14:33:07 2018 us=353351 Rumbles/10.10.10.139:57314 Data Channel: using negotiated cipher 'AES-256-GCM'
Wed Mar 28 14:33:07 2018 us=353566 Rumbles/10.10.10.139:57314 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Mar 28 14:33:07 2018 us=353589 Rumbles/10.10.10.139:57314 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key

TinCanTech

Rumbles wrote making those changes, but will they actually improve the number of drops we are seeing?

I believe by disabling NCP the particular error you are seeing should go away completely (at least for 99% of connection) but that is only my personal experience not an official answer ..

Rumbles wrote Are you saying NCP is a likely cause for the drops and disabling it might help?

I believe so ..

Rumbles wrote But if I do that I should change ciphers at the same time

Yes .. because your current cipher is blowfish (the default)

Rumbles wrote If I change to AES-256-CBC do I need to change the reneg-sec setting also?

I would recommend not using --reneg 0 ever .. unless you implement one of the other reneg parameters.

Rumbles wrote Is there any way to check whether NCP has kicked in

It is on by default (ovpn 2.4.x) , disable with --ncp-disable .. and check your logs.

Rumbles

TinCanTech wrote
Rumbles wrote If I change to AES-256-CBC do I need to change the reneg-sec setting also?
I would recommend not using --reneg 0 ever .. unless you implement one of the other reneg parameters.

Thanks again, can you clarify what other reneg parameters you can use? I have been told by a colleague that they need reneg set to 0 (as we use the Duo plugin for 2FA and they recommend it), so I could do with having a discussion with them about it, but would want to know what parameters you are referring to.

TinCanTech

Rumbles wrote what other reneg parameters you can use?

See --reneg-* in The Manual v24x

Rumbles wrote I have been told by a colleague that they need reneg set to 0 (as we use the Duo plugin for 2FA and they recommend it), so I could do with having a discussion with them about it

2FA does not protect against SWEET32 ..

Disabling features because they are inconvenient is usually (and in this case) a bad idea ..

You may want to look into --auth-gen-token & --auth-token

Rumbles

Thanks, we were just going with what Duo recommended. I believe they recommend this, as every time you get a renegotiation another request gets pushed out to the device, which might be missed.

I just had someone connect and I saw this in the logs:

WARNING: cipher with small block size in use, reducing reneg-bytes to 64MB to mitigate SWEET32 attacks.

So I guess the server is seeing that we have a small block size on our cipher and making this change to cover us... That could also explain why someone was seeing drops every time he tried to transfer large files. I guess changing to a better cipher would fix this issue for us.