TLS Error when more than 1000 clients

[Coconuts]

Hello,

I've been searching everywhere but unable to find the answer.

I've got a backdoor server that is use for ssh for debug on a park at various clients sites.
The server is running well and use almost no CPU, but recently we've reach more than 1000 clients so I up-scaled the server.
At first using max client but I got the "too many files" error that i fixed by allowing OpenVPN to open more and that worked fine.
But now, when reaching 1000 clients i got the error that I cannot fix :

X.X.X.X:46404 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
X.X.X.X:46404 TLS Error: TLS handshake failed
X.X.X.X:46404 Fatal TLS error (check_tls_errors_co), restarting
X.X.X.X:46404 SIGUSR1[soft,tls-error] received, client-instance restarting

Hopping you can help me,
Thank you

[TinCanTech]

--max-clients defaults to 1024, perhaps that is the issue.

[Coconuts]

Thanks for the answer, but if got it to 4096 but still the same problem

[TinCanTech]

viewtopic.php?t=22603

[Coconuts]

Hello again,

I use OpenVPN 2.4.7 running of ubuntu 20.04 and here is the server config :

View OriginalServer Config

# Which local IP address should OpenVPN
# listen on? (optional)
;local a.b.c.d

# Which TCP/UDP port should OpenVPN listen on?
# If you want to run multiple OpenVPN instances
# on the same machine, use a different port
# number for each one. You will need to
# open up this port on your firewall.
;port 1194
port 443

# TCP or UDP server?
proto tcp
;proto udp

# "dev tun" will create a routed IP tunnel,
# "dev tap" will create an ethernet tunnel.
# Use "dev tap0" if you are ethernet bridging
# and have precreated a tap0 virtual interface
# and bridged it with your ethernet interface.
# If you want to control access policies
# over the VPN, you must create firewall
# rules for the the TUN/TAP interface.
# On non-Windows systems, you can give
# an explicit unit number, such as tun0.
# On Windows, use "dev-node" for this.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
;dev tap
dev tun0

# Windows needs the TAP-Win32 adapter name
# from the Network Connections panel if you
# have more than one. On XP SP2 or higher,
# you may need to selectively disable the
# Windows firewall for the TAP adapter.
# Non-Windows systems usually don't need this.
;dev-node MyTap

# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key). Each client
# and the server must have their own cert and
# key file. The server and all clients will
# use the same ca file.
#
# See the "easy-rsa" directory for a series
# of scripts for generating RSA certificates
# and private keys. Remember to use
# a unique Common Name for the server
# and each of the client certificates.
#
# Any X509 key management system can be used.
# OpenVPN can also use a PKCS #12 formatted key file
# (see "pkcs12" directive in man page).
ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/serverVPN.crt
key /etc/openvpn/server/serverVPN.key # This file should be kept secret

# Diffie hellman parameters.
# Generate your own with:
# openssl dhparam -out dh2048.pem 2048
;dh dh2048.pem
dh none

# Network topology
# Should be subnet (addressing via IP)
# unless Windows clients v2.0.9 and lower have to
# be supported (then net30, i.e. a /30 per client)
# Defaults to net30 (not recommended)
topology subnet

# Configure server mode and supply a VPN subnet
# for OpenVPN to draw client addresses from.
# The server will take 10.8.0.1 for itself,
# the rest will be made available to clients.
# Each client will be able to reach the server
# on 10.8.0.1. Comment this line out if you are
# ethernet bridging. See the man page for more info.
server 10.8.0.0 255.255.240.0

# Maintain a record of client <-> virtual IP address
# associations in this file. If OpenVPN goes down or
# is restarted, reconnecting clients can be assigned
# the same virtual IP address from the pool that was
# previously assigned.
ifconfig-pool-persist /var/log/openvpn/ipp.txt

# Configure server mode for ethernet bridging.
# You must first use your OS's bridging capability
# to bridge the TAP interface with the ethernet
# NIC interface. Then you must manually set the
# IP/netmask on the bridge interface, here we
# assume 10.8.0.4/255.255.255.0. Finally we
# must set aside an IP range in this subnet
# (start=10.8.0.50 end=10.8.0.100) to allocate
# to connecting clients. Leave this line commented
# out unless you are ethernet bridging.
;server-bridge 10.8.0.4 255.255.255.0 10.8.0.50 10.8.0.100

# Configure server mode for ethernet bridging
# using a DHCP-proxy, where clients talk
# to the OpenVPN server-side DHCP server
# to receive their IP address allocation
# and DNS server addresses. You must first use
# your OS's bridging capability to bridge the TAP
# interface with the ethernet NIC interface.
# Note: this mode only works on clients (such as
# Windows), where the client-side TAP adapter is
# bound to a DHCP client.
;server-bridge

# Push routes to the client to allow it
# to reach other private subnets behind
# the server. Remember that these
# private subnets will also need
# to know to route the OpenVPN client
# address pool (10.8.0.0/255.255.255.0)
# back to the OpenVPN server.
;push "route 10.0.0.0 255.255.0.0"
;push "route 192.168.20.0 255.255.255.0"

# To assign specific IP addresses to specific
# clients or if a connecting client has a private
# subnet behind it that should also have VPN access,
# use the subdirectory "ccd" for client-specific
# configuration files (see man page for more info).

# EXAMPLE: Suppose the client
# having the certificate common name "Thelonious"
# also has a small subnet behind his connecting
# machine, such as 192.168.40.128/255.255.255.248.
# First, uncomment out these lines:
;client-config-dir /etc/openvpn/ccd
;route 192.168.40.128 255.255.255.248
# Then create a file ccd/Thelonious with this line:
# iroute 192.168.40.128 255.255.255.248
# This will allow Thelonious' private subnet to
# access the VPN. This example will only work
# if you are routing, not bridging, i.e. you are
# using "dev tun" and "server" directives.

# EXAMPLE: Suppose you want to give
# Thelonious a fixed VPN IP address of 10.9.0.1.
# First uncomment out these lines:
;client-config-dir ccd
;route 10.9.0.0 255.255.255.252
# Then add this line to ccd/Thelonious:
# ifconfig-push 10.9.0.1 10.9.0.2

# Suppose that you want to enable different
# firewall access policies for different groups
# of clients. There are two methods:
# (1) Run multiple OpenVPN daemons, one for each
# group, and firewall the TUN/TAP interface
# for each group/daemon appropriately.
# (2) (Advanced) Create a script to dynamically
# modify the firewall in response to access
# from different clients. See man
# page for more info on learn-address script.
;learn-address ./script

# If enabled, this directive will configure
# all clients to redirect their default
# network gateway through the VPN, causing
# all IP traffic such as web browsing and
# and DNS lookups to go through the VPN
# (The OpenVPN server machine may need to NAT
# or bridge the TUN/TAP interface to the internet
# in order for this to work properly).
;push "redirect-gateway def1 bypass-dhcp"

# Certain Windows-specific network settings
# can be pushed to clients, such as DNS
# or WINS server addresses. CAVEAT:
# http://openvpn.net/faq.html#dhcpcaveats
# The addresses below refer to the public
# DNS servers provided by opendns.com.
;push "dhcp-option DNS 208.67.222.222"
;push "dhcp-option DNS 208.67.220.220"

# Uncomment this directive to allow different
# clients to be able to "see" each other.
# By default, clients will only see the server.
# To force clients to only see the server, you
# will also need to appropriately firewall the
# server's TUN/TAP interface.
client-to-client

# Uncomment this directive if multiple clients
# might connect with the same certificate/key
# files or common names. This is recommended
# only for testing purposes. For production use,
# each client should have its own certificate/key
# pair.
#
# IF YOU HAVE NOT GENERATED INDIVIDUAL
# CERTIFICATE/KEY PAIRS FOR EACH CLIENT,
# EACH HAVING ITS OWN UNIQUE "COMMON NAME",
# UNCOMMENT THIS LINE OUT.
duplicate-cn

# The keepalive directive causes ping-like
# messages to be sent back and forth over
# the link so that each side knows when
# the other side has gone down.
# Ping every 10 seconds, assume that remote
# peer is down if no ping received during
# a 120 second time period.
keepalive 120 500

# For extra security beyond that provided
# by SSL/TLS, create an "HMAC firewall"
# to help block DoS attacks and UDP port flooding.
#
# Generate with:
# openvpn --genkey --secret ta.key
#
# The server and each client must have
# a copy of this key.
# The second parameter should be '0'
# on the server and '1' on the clients.
;tls-auth ta.key 0 # This file is secret
tls-crypt /etc/openvpn/server/ta.key

# Select a cryptographic cipher.
# This config item must be copied to
# the client config file as well.
# Note that v2.4 client/server will automatically
# negotiate AES-256-GCM in TLS mode.
# See also the ncp-cipher option in the manpage
cipher AES-256-CBC
;cipher AES-256-GCM

;data-ciphers AES-256-GCM:AES-256-CBC:AES-128-GCM:AES-128-CBC:BF-CBC

auth SHA256

# Enable compression on the VPN link and push the
# option to the client (v2.4+ only, for earlier
# versions see below)
;compress lz4-v2
;push "compress lz4-v2"

# For compression compatible with older clients use comp-lzo
# If you enable it here, you must also
# enable it in the client config file.
;comp-lzo

# The maximum number of concurrently connected
# clients we want to allow.
max-clients 4096

# It's a good idea to reduce the OpenVPN
# daemon's privileges after initialization.
#
# You can uncomment this out on
# non-Windows systems.
user nobody
group nogroup

# The persist options will try to avoid
# accessing certain resources on restart
# that may no longer be accessible because
# of the privilege downgrade.
persist-key
persist-tun

# Output a short status file showing
# current connections, truncated
# and rewritten every minute.
status /var/log/openvpn/openvpn-status.log

# By default, log messages will go to the syslog (or
# on Windows, if running as a service, they will go to
# the "\Program Files\OpenVPN\log" directory).
# Use log or log-append to override this default.
# "log" will truncate the log file on OpenVPN startup,
# while "log-append" will append to it. Use one
# or the other (but not both).
log /var/log/openvpn/openvpn.log
;log-append /var/log/openvpn/openvpn.log

# Set the appropriate level of log
# file verbosity.
#
# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 3

# Silence repeating messages. At most 20
# sequential messages of the same message
# category will be output to the log.
;mute 20

# Notify the client that when the server restarts so it
# can automatically reconnect.
;explicit-exit-notify 1
explicit-exit-notify 0

Thanks

[openvpn_inc]

Hi Coco,

I would wonder, is this happening to every client in excess of a certain limit, or just some clients?

In the latter case, this is something we commonly see with larger Access Server or OpenVPN Cloud deployments, because some users are using bad ISPs (or low-quality home routers, often provided by bad ISPs) which don't properly support outbound UDP connections.

In the former case, I'd investigate if there are issues with limits in your own router (the one in front of the OpenVPN server.)

What you might do, just as AS/Cloud both do: run a separate daemon with a different (adjacent CIDR is best) netblock, running TCP protocol. TCP port 443 (https) is best. If you already have an https server on the OpenVPN machine, see --port-share in the manual. (I see you're using 2.4 not 2.5, so use your own local "man openvpn", or hunt down the online 2.4 manual -- it is at the community wiki.)

To allow clients to use either TCP or UDP, use multiple --remote lines in the client configuration file:

Code: Select all

remote vpn.example.org 1194 udp
remote vpn.example.org 1194 udp
remote vpn.example.org 443 tcp
remote vpn.example.org 1194 udp

This is how AS/Cloud generate client profiles. The client would try 1194/udp twice, then 443/tcp once, then 1194/udp once again.

While it would be a substantial expense for your organization, I would also suggest that you check out the commercial product: Access Server and/or the OpenVPN Cloud managed service. These handle a lot of the more challenging issues of VPN service, as well as adding a lot of user and admin convenience. A small-scale deployment is free of charge, and you can also get a trial license for proof of concept, if you ask at the Support link in my signature.

Another benefit is that AS works around the issue of openvpn being a single-threaded application: it binds a separate daemon to each CPU core, so as to spread out the CPU load. This is a common problem with larger open source openvpn server deployments.

(Sorry for the sales pitch: we are indeed here to support the open source openvpn. But generally large users are happier with AS/Cloud, so I thought it was worth mentioning. Do feel free to ignore it.)

regards, rob0

[TinCanTech]

I asked for server logs, none were provided.

My crystal ball only works on Friday 13.

[openvpn_inc]

My crystal ball only works on Friday 13.

Hi TCT,

I'm looking forward to 2023-01-13!

regards, rob0

[TinCanTech]

I asked for server logs, none were provided.

My crystal ball only works on Friday 13.

My crystal ball only works on Friday 13.

Hi TCT,

I'm looking forward to 2023-01-13!

regards, rob0

Hi there rob0

I never claimed it was accurate

[TinCanTech]

This is most interesting:

TZero Time: Coconuts » 2022 Jul 13 : 13:03

Delta Time: TinCanTech » 2022 Jul 20 : 22:20

Friday 13 Time: Universal » 2023 Jan 13 :

The Matrix is everywhere, even in this very thread