Weird limit, no route to host until I ping from client

[PöpecRob]

Hi,

Can someone help me?

I have a debian 9 and from Feb. 16 we updatated openvpn server to OpenVPN 2.4.7. We have nearly 340 clients on this server, but from this update (But it is not sure that it is caused by the update) when we reach the 254 clients connected, all following clients are able to connect too, but the communication from the server to client is not working, I mean I cannot ping, or SSH to the clients. It writes me no route to host, or destination host unreachable. If I ping the server from the clients, the communication is starts to work both ways right away and stays working until it's connected. (But if I do not ping from the client, there is no way to initialize the connection from the server to the client)

Before everything was worked without any issues. And we haven't made any changes to the clients, only the server.

Do anyone have an idea why is it doing this?
(I was thinking that it might be caused by ipv6, or something, but I am not sure)

My config is looks like this:

Code: Select all

local ....
port 443
proto udp
dev tap
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert server.crt
key server.key
dh dh2048.pem
script-security 2
crl-verify easy-rsa/keys/crl.pem
#tls-verify "/usr/local/sbin/ovpnCNcheck.py /etc/openvpn/CN_whitelist"
server 10.9.0.0 255.255.0.0
ifconfig-pool-persist ipp.txt
# push "redirect-gateway def1"
# push "dhcp-option WINS 10.9.0.1"
# push "dhcp-option DNS 8.8.8.8"
# push "dhcp-option DNS 8.8.4.4"
client-config-dir /etc/openvpn/static
#client-to-client
keepalive 10 120
persist-key
persist-tun
status openvpn-status.log
log-append openvpn-connects.log
management localhost 7505
verb 3

Client config is looks like this:

Code: Select all

client
proto udp
dev tap
remote .... 443
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert .....crt
key .....key
remote-cert-tls server
verb 3
mute 20
ping-restart 10
tls-version-min 1.0

[TinCanTech]

we updatated openvpn server to OpenVPN 2.4.7.

What version was it before update ?

Also, please add your server and client log at the time the problem occurs.
( with 340 clients that may be difficult)

[PöpecRob]

Thank you for your answer!

I cannot tell you the old version of OpenVPN, but I am on it.

Here is the log when the connection is not working. If I ping the server from client, it won't append. Neither the server or client log.

Code: Select all

Server:
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 TLS: Initial packet from [AF_INET]{ipaddress}:45565, sid=74634ec4 6932e2bb
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 VERIFY OK: depth=1, C=HU, ST=City, L=City, O=Company Computer, OU=CompanyCloud, CN=Company Computer CA, name=CompanyCloud, emailAddress=info@notexistingdomainp.com
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 VERIFY OK: depth=0, C=HU, ST=City, L=City, O=Company Computer, OU=CompanyCloud, CN=02c000816ee14e19, name=CompanyCloud, emailAddress=info@notexistingdomainp.com
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_VER=2.4.7
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_PLAT=linux
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_PROTO=2
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_NCP=2
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_LZ4=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_LZ4v2=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_LZO=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_COMP_STUB=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_COMP_STUBv2=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 peer info: IV_TCPNL=1
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Wed Apr  6 01:13:18 2022 {ipaddress}:45565 [02c000816ee14e19] Peer Connection Initiated with [AF_INET]{ipaddress}:45565
Wed Apr  6 01:13:18 2022 MULTI: new connection by client '02c000816ee14e19' will cause previous active sessions by this client to be dropped.  Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect.
Wed Apr  6 01:13:18 2022 MULTI_sva: pool returned IPv4=10.9.2.176, IPv6=(Not enabled)
Wed Apr  6 01:13:20 2022 02c000816ee14e19/{ipaddress}:45565 PUSH: Received control message: 'PUSH_REQUEST'
Wed Apr  6 01:13:20 2022 02c000816ee14e19/{ipaddress}:45565 SENT CONTROL [02c000816ee14e19]: 'PUSH_REPLY,route-gateway 10.9.0.1,ping 10,ping-restart 120,ifconfig 10.9.2.176 255.255.0.0,peer-id 82,cipher AES-256-GCM' (status=1)
Wed Apr  6 01:13:20 2022 02c000816ee14e19/{ipaddress}:45565 Data Channel Encrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Apr  6 01:13:20 2022 02c000816ee14e19/{ipaddress}:45565 Data Channel Decrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Apr  6 01:13:20 2022 02c000816ee14e19/{ipaddress}:45565 MULTI: Learn: ea:a3:3d:e9:67:81 -> 02c000816ee14e19/{ipaddress}:45565


Client:
Tue Apr  5 23:13:17 2022 OpenVPN 2.4.7 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 28 2021
Tue Apr  5 23:13:17 2022 library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
Tue Apr  5 23:13:18 2022 TCP/UDP: Preserving recently used remote address: [AF_INET]{serveripaddress}:443
Tue Apr  5 23:13:18 2022 Socket Buffers: R=[180224->180224] S=[180224->180224]
Tue Apr  5 23:13:18 2022 UDP link local: (not bound)
Tue Apr  5 23:13:18 2022 UDP link remote: [AF_INET]{serveripaddress}:443
Tue Apr  5 23:13:18 2022 TLS: Initial packet from [AF_INET]{serveripaddress}:443, sid=3745ade7 194c045b
Tue Apr  5 23:13:18 2022 VERIFY OK: depth=1, C=HU, ST=City, L=City, O=Company Computer, OU=CompanyCloud, CN=Company Computer CA, name=CompanyCloud, emailAddress=info@notexistingdomainp.com
Tue Apr  5 23:13:18 2022 VERIFY KU OK
Tue Apr  5 23:13:18 2022 Validating certificate extended key usage
Tue Apr  5 23:13:18 2022 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Tue Apr  5 23:13:18 2022 VERIFY EKU OK
Tue Apr  5 23:13:18 2022 VERIFY OK: depth=0, C=HU, ST=City, L=City, O=Company Computer, OU=CompanyCloud, CN=server, name=CompanyCloud, emailAddress=info@notexistingdomainp.com
Tue Apr  5 23:13:18 2022 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Tue Apr  5 23:13:18 2022 [server] Peer Connection Initiated with [AF_INET]{serveripaddress}:443
Tue Apr  5 23:13:19 2022 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Tue Apr  5 23:13:19 2022 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.9.0.1,ping 10,ping-restart 120,ifconfig 10.9.2.176 255.255.0.0,peer-id 82,cipher AES-256-GCM'
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: timers and/or timeouts modified
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: --ifconfig/up options modified
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: route-related options modified
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: peer-id set
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: adjusting link_mtu to 1656
Tue Apr  5 23:13:19 2022 OPTIONS IMPORT: data channel crypto options modified
Tue Apr  5 23:13:19 2022 Data Channel: using negotiated cipher 'AES-256-GCM'
Tue Apr  5 23:13:19 2022 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Tue Apr  5 23:13:19 2022 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Tue Apr  5 23:13:19 2022 TUN/TAP device tap0 opened
Tue Apr  5 23:13:19 2022 TUN/TAP TX queue length set to 100
Tue Apr  5 23:13:19 2022 /sbin/ip link set dev tap0 up mtu 1500
Tue Apr  5 23:13:19 2022 /sbin/ip addr add dev tap0 10.9.2.176/16 broadcast 10.9.255.255
Tue Apr  5 23:13:19 2022 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Tue Apr  5 23:13:19 2022 Initialization Sequence Completed

[PöpecRob]

I've found that from client's arp this line is missing, until I ping from client to server.

Code: Select all

10.9.0.1                 ether   ba:14:ed:0a:a4:2e   C                     tap0

Also from client's route table this line is missing until I ping from client to server:

Code: Select all

link-local      0.0.0.0         255.255.0.0     U     213    0        0 tap0

Arp

Code: Select all

Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.115            ether   92:b9:a3:49:91:fd   C                     eth0
192.168.1.146            ether   e2:06:b9:5b:93:75   C                     eth0
192.168.1.237            ether   0a:90:45:b1:f3:04   C                     eth0
192.168.1.107            ether   5c:e4:2a:21:35:c7   C                     eth0
10.9.0.1                 ether   ba:14:ed:0a:a4:2e   C                     tap0
_gateway                 ether   e8:48:b8:42:07:e8   C                     eth0
192.168.1.11             ether   dc:56:e7:57:68:74   C                     eth0

Full routing table

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    202    0        0 eth0
10.9.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tap0
link-local      0.0.0.0         255.255.0.0     U     213    0        0 tap0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     202    0        0 eth0

[PöpecRob]

Is it possible, that it is a lower level problem in kernel/driver with the new update?