New OpenVPN Version breaks OS routes

Akito

OpenVPN was used inside Docker containers running Debian stable for a long time.

At some point, there was an attempt to upgrade the Debian versions running inside the containers.

The version used before:
Debian 10 Buster -> openvpn (2.4.7-1+deb10u1) -> https://packages.debian.org/buster/openvpn

The version after the upgrade:
Debian 11 Bullseye -> openvpn (2.5.1-3) -> https://packages.debian.org/bullseye/openvpn

After upgrading, OpenVPN started to break OS routes on installation.

Example of working routes, how they were at

openvpn (2.4.7-1+deb10u1)

and should be:

default via 10.35.0.1 dev eth0 
10.35.0.0/24 dev eth0 proto kernel scope link src 10.35.0.195 
10.35.0.0/16 via 10.35.0.1 dev eth0 
10.190.0.0/16 via 10.190.0.12 dev tun0

Example of broken routes, how they are at

openvpn (2.5.1-3)

and should not be:

default via 10.35.0.1 dev eth0 
10.35.0.0/24 via 10.190.0.12 dev tun0 
10.35.0.0/16 via 10.35.0.1 dev eth0 
10.190.0.0/16 via 10.190.0.12 dev tun0

I installed an even newer 2.5.x version and the issue was still not solved.

I am out of ideas why this might happen. It worked great before and now it is broken for several 2.5.x versions.
Neither was I able to find any relevant change regarding this functionality in the list of commits, provided as some kind of changelog.

The only commit that was related to gateway routing semed not to be relevant, as the behaviour did not change when changing to versions that did not incorporate that commit.

server 10.190.0.0 255.255.0.0
verb 3
client-to-client
crl-verify /path/to/pki/crl.pem
key /path/to/pki/private/server.key
ca /path/to/pki/ca.crt
cert /path/to/pki/issued/server.crt
dh /path/to/pki/dh.pem
tls-auth /path/to/pki/ta.key 0
cipher AES-256-GCM
key-direction 0
keepalive 10 60
persist-key
persist-tun
proto tcp
port 4443
dev tun0
status /var/log/openvpn-status.log
user nobody
group nogroup
client-config-dir /path/to/config/ccd
push "route 10.35.0.0 255.255.255.0"
push "dhcp-option DOMAIN-SEARCH debian-11-openvpn-1.svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH cluster.local"
push "dhcp-option DOMAIN-SEARCH invalid"

push "dhcp-option DNS 10.36.0.10"

One of the many clients' config:

nobind
dev tun
remote openvpn 4443 tcp
cipher AES-256-GCM
<key>
# Private Key
</key>
<cert>
# Certificate
</cert>
<ca>
# CA Certificate
</ca>
<tls-auth>
# Static Key
</tls-auth>
key-direction 1

CCD config for this particular client:

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

Client log is unavailable and server log does not show anything beyond normal behaviour. The issue arises 100% client-side. The server does not even use Debian, in the first place.

Server OpenVPN version information:

OpenVPN 2.4.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push='no' enable_comp_stub='no' enable_crypto='yes' enable_crypto_ofb_cfb='yes' enable_debug='yes' enable_def_auth='yes' enable_dlopen='unknown' enable_dlopen_self='unknown' enable_dlopen_self_static='unknown' enable_fast_install='yes' enable_fragment='yes' enable_iproute2='yes' enable_libtool_lock='yes' enable_lz4='yes' enable_lzo='yes' enable_management='yes' enable_multihome='yes' enable_pam_dlopen='no' enable_pedantic='no' enable_pf='yes' enable_pkcs11='no' enable_plugin_auth_pam='yes' enable_plugin_down_root='yes' enable_plugins='yes' enable_port_share='yes' enable_selinux='no' enable_server='yes' enable_shared='yes' enable_shared_with_static_runtimes='no' enable_small='no' enable_static='yes' enable_strict='no' enable_strict_options='no' enable_systemd='no' enable_werror='no' enable_win32_dll='yes' enable_x509_alt_username='no' with_aix_soname='aix' with_crypto_library='openssl' with_gnu_ld='yes' with_mem_check='no' with_sysroot='no'

TinCanTech

Please see https://forums.openvpn.net/viewtopic.php?f=30&t=22603#p68963

Akito

TinCanTech wrote Please see https://forums.openvpn.net/viewtopic.php?f=30&t=22603#p68963

@TinCanTech

Additional information was added to the original post.

TinCanTech

No logs ..

Anyway, it looks like you forgot --topology subnet, see the manual.

Akito

TinCanTech wrote No logs ..

Anyway, it looks like you forgot --topology subnet, see the manual.

My assumption is, that nothing is configured wrongly or "forgotten", as it worked before, for a long time. The only thing that breaks it is the new OpenVPN version. So, it must be an issue with OpenVPN.

I am looking up that option and will check if this workaround can be successfully applied. However, the issue should be fixed in OpenVPN, not worked around by adding configuration options.

TinCanTech

TinCanTech wrote No logs ..

:roll:

                msg(M_WARN, "WARNING: --topology net30 support for server "
                    "configs with IPv4 pools will be removed in a future "
                    "release. Please migrate to --topology subnet as soon "
                    "as possible.");

Akito

TinCanTech wrote Anyway, it looks like you forgot --topology subnet, see the manual.

@TinCanTech

Adding that option to the server config broke everything, because no routes were pushed, at all. So, a connection could not be established, at all. This issue applies to both, the working and the broken client versions.

The default

topology net30

option at least makes the older, working, client work, whereas breaks the routes on the newer, broken, clients.

I've added a CCD config to the original post, as it is used for all clients and proven to work with older, working, client versions.

Akito

TinCanTech wrote

TinCanTech wrote No logs ..

:roll:

                msg(M_WARN, "WARNING: --topology net30 support for server "
                    "configs with IPv4 pools will be removed in a future "
                    "release. Please migrate to --topology subnet as soon "
                    "as possible.");

@TinCanTech

I've grepped through the entire server log and this message does not appear once. Seems like this is indeed an issue with OpenVPN, after all. 😉

Akito

@TinCanTech

Anyway, thanks for showing the source code. This issue has to be fixed in OpenVPN. The message literally does not show up anywhere, in any log I had available. If this message would've been shown, I could've probably found the issue myself.

Thank you very much!

TinCanTech

Akito wrote CCD config for this particular client:

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

That is completely incorrect setting for --ifconfig-push, which explains why it doesn't work.

Pippin

Hi,

This is wrong:

Akito wrote CCD config for this particular client:

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

It may have worked but that's more due to the quirks of net30...

This is right:
client1

ifconfig-push 10.190.0.6 10.190.0.5

client2

ifconfig-push 10.190.0.10 10.190.0.9

client3

ifconfig-push 10.190.0.14 10.190.0.13

etc.

But really, now could be the time to migrate to topology subnet, it's not that difficult and is easier to understand.
.

Akito

Pippin wrote
But really, now could be the time to migrate to topology subnet, it's not that difficult and is easier to understand.

I am in the process of doing so. During that process I have found the following information.

https://serverfault.com/a/867558/405521

According to that answer, your suggestions won't work for

topology subnet

.

As it is now, I am currently trying to make

topology subnet

work by debugging connectivity issues arising client-side.

Pippin

With topology subnet the server will listen on .1
There will be room for 65532 clients in a /16 (subnet size minus four).
You really need that much?

The --ifconfig-push directive changes to,

ifconfig-push 10.190.0.2 255.255.0.0

The client would get the .2 address.

Always read the manual first,
https://build.openvpn.net/man/openvpn-2.5/openvpn.8.html
and Howto,
https://community.openvpn.net/openvpn/wiki/HOWTO

Akito

I consider this solved now, as the `net30` vs `subnet` situation has been worked out.

There is only one step left: https://forums.openvpn.net/viewtopic.php?f=6&t=33734#p104984

Would appreciate support for letting me complete this final step.