New OpenVPN Version breaks OS routes

[Akito]

OpenVPN was used inside Docker containers running Debian stable for a long time.

At some point, there was an attempt to upgrade the Debian versions running inside the containers.

The version used before:
Debian 10 Buster -> openvpn (2.4.7-1+deb10u1) -> https://packages.debian.org/buster/openvpn

The version after the upgrade:
Debian 11 Bullseye -> openvpn (2.5.1-3) -> https://packages.debian.org/bullseye/openvpn

After upgrading, OpenVPN started to break OS routes on installation.

Example of working routes, how they were at

Code: Select all

openvpn (2.4.7-1+deb10u1)

and should be:

Code: Select all

default via 10.35.0.1 dev eth0 
10.35.0.0/24 dev eth0 proto kernel scope link src 10.35.0.195 
10.35.0.0/16 via 10.35.0.1 dev eth0 
10.190.0.0/16 via 10.190.0.12 dev tun0

Example of broken routes, how they are at

Code: Select all

openvpn (2.5.1-3)

and should not be:

Code: Select all

default via 10.35.0.1 dev eth0 
10.35.0.0/24 via 10.190.0.12 dev tun0 
10.35.0.0/16 via 10.35.0.1 dev eth0 
10.190.0.0/16 via 10.190.0.12 dev tun0 

I installed an even newer 2.5.x version and the issue was still not solved.

I am out of ideas why this might happen. It worked great before and now it is broken for several 2.5.x versions.
Neither was I able to find any relevant change regarding this functionality in the list of commits, provided as some kind of changelog.

The only commit that was related to gateway routing semed not to be relevant, as the behaviour did not change when changing to versions that did not incorporate that commit.

View Originalserver

server 10.190.0.0 255.255.0.0
verb 3
client-to-client
crl-verify /path/to/pki/crl.pem
key /path/to/pki/private/server.key
ca /path/to/pki/ca.crt
cert /path/to/pki/issued/server.crt
dh /path/to/pki/dh.pem
tls-auth /path/to/pki/ta.key 0
cipher AES-256-GCM
key-direction 0
keepalive 10 60
persist-key
persist-tun
proto tcp
port 4443
dev tun0
status /var/log/openvpn-status.log
user nobody
group nogroup
client-config-dir /path/to/config/ccd
push "route 10.35.0.0 255.255.255.0"
push "dhcp-option DOMAIN-SEARCH debian-11-openvpn-1.svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH cluster.local"
push "dhcp-option DOMAIN-SEARCH invalid"

push "dhcp-option DNS 10.36.0.10"

One of the many clients' config:

View Originalclient

nobind
dev tun
remote openvpn 4443 tcp
cipher AES-256-GCM
<key>
# Private Key
</key>
<cert>
# Certificate
</cert>
<ca>
# CA Certificate
</ca>
<tls-auth>
# Static Key
</tls-auth>
key-direction 1

CCD config for this particular client:

Code: Select all

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

Client log is unavailable and server log does not show anything beyond normal behaviour. The issue arises 100% client-side. The server does not even use Debian, in the first place.

Server OpenVPN version information:

Code: Select all

OpenVPN 2.4.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push='no' enable_comp_stub='no' enable_crypto='yes' enable_crypto_ofb_cfb='yes' enable_debug='yes' enable_def_auth='yes' enable_dlopen='unknown' enable_dlopen_self='unknown' enable_dlopen_self_static='unknown' enable_fast_install='yes' enable_fragment='yes' enable_iproute2='yes' enable_libtool_lock='yes' enable_lz4='yes' enable_lzo='yes' enable_management='yes' enable_multihome='yes' enable_pam_dlopen='no' enable_pedantic='no' enable_pf='yes' enable_pkcs11='no' enable_plugin_auth_pam='yes' enable_plugin_down_root='yes' enable_plugins='yes' enable_port_share='yes' enable_selinux='no' enable_server='yes' enable_shared='yes' enable_shared_with_static_runtimes='no' enable_small='no' enable_static='yes' enable_strict='no' enable_strict_options='no' enable_systemd='no' enable_werror='no' enable_win32_dll='yes' enable_x509_alt_username='no' with_aix_soname='aix' with_crypto_library='openssl' with_gnu_ld='yes' with_mem_check='no' with_sysroot='no'

[TinCanTech]

Please see viewtopic.php?f=30&t=22603#p68963

[Akito]

Please see viewtopic.php?f=30&t=22603#p68963

@TinCanTech

Additional information was added to the original post.

[TinCanTech]

No logs ..

Anyway, it looks like you forgot --topology subnet, see the manual.

[Akito]

No logs ..

Anyway, it looks like you forgot --topology subnet, see the manual.

My assumption is, that nothing is configured wrongly or "forgotten", as it worked before, for a long time. The only thing that breaks it is the new OpenVPN version. So, it must be an issue with OpenVPN.

I am looking up that option and will check if this workaround can be successfully applied. However, the issue should be fixed in OpenVPN, not worked around by adding configuration options.

[TinCanTech]

No logs ..

Code: Select all

                msg(M_WARN, "WARNING: --topology net30 support for server "
                    "configs with IPv4 pools will be removed in a future "
                    "release. Please migrate to --topology subnet as soon "
                    "as possible.");

[Akito]

Anyway, it looks like you forgot --topology subnet, see the manual.

@TinCanTech

Adding that option to the server config broke everything, because no routes were pushed, at all. So, a connection could not be established, at all. This issue applies to both, the working and the broken client versions.

The default

Code: Select all

topology net30

option at least makes the older, working, client work, whereas breaks the routes on the newer, broken, clients.

I've added a CCD config to the original post, as it is used for all clients and proven to work with older, working, client versions.

[Akito]

No logs ..

Code: Select all

                msg(M_WARN, "WARNING: --topology net30 support for server "
                    "configs with IPv4 pools will be removed in a future "
                    "release. Please migrate to --topology subnet as soon "
                    "as possible.");

@TinCanTech

I've grepped through the entire server log and this message does not appear once. Seems like this is indeed an issue with OpenVPN, after all.

[Akito]

@TinCanTech

Anyway, thanks for showing the source code. This issue has to be fixed in OpenVPN. The message literally does not show up anywhere, in any log I had available. If this message would've been shown, I could've probably found the issue myself.

Thank you very much!

[TinCanTech]

CCD config for this particular client:

Code: Select all

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

That is completely incorrect setting for --ifconfig-push, which explains why it doesn't work.

[Pippin]

Hi,

This is wrong:

CCD config for this particular client:

Code: Select all

ifconfig-push 10.190.0.12 10.190.0.12

The purpose of this CCD config is to apply a static IP to that particular client.

It may have worked but that's more due to the quirks of net30...

This is right:
client1

Code: Select all

ifconfig-push 10.190.0.6 10.190.0.5

client2

Code: Select all

ifconfig-push 10.190.0.10 10.190.0.9

client3

Code: Select all

ifconfig-push 10.190.0.14 10.190.0.13

etc.

But really, now could be the time to migrate to topology subnet, it's not that difficult and is easier to understand.
.

[Akito]

But really, now could be the time to migrate to topology subnet, it's not that difficult and is easier to understand.

I am in the process of doing so. During that process I have found the following information.

https://serverfault.com/a/867558/405521

According to that answer, your suggestions won't work for

Code: Select all

topology subnet

.

As it is now, I am currently trying to make

Code: Select all

topology subnet

work by debugging connectivity issues arising client-side.

[Pippin]

With topology subnet the server will listen on .1
There will be room for 65532 clients in a /16 (subnet size minus four).
You really need that much?

The --ifconfig-push directive changes to,

Code: Select all

ifconfig-push 10.190.0.2 255.255.0.0

The client would get the .2 address.

Always read the manual first,
https://build.openvpn.net/man/openvpn-2 ... vpn.8.html
and Howto,
https://community.openvpn.net/openvpn/wiki/HOWTO

[Akito]

I consider this solved now, as the `net30` vs `subnet` situation has been worked out.

There is only one step left: viewtopic.php?f=6&t=33734#p104984

Would appreciate support for letting me complete this final step.