OpenVPN AS 2.11.3 seems broken

alex.ong

I setup a Fresh VM on Ubuntu 22.04 to install 2.11.3.
I was previously using Ubuntu 20.04 with OpenVPN 2.8.7

1) I installed it successfully, and it directed me to https://10.0.1.86:943/
* This works fine; however the normal https://10.0.1.86/ url should work and redirect via daemon
* The settings are the same as my 2.8.7 install (443TCP/1194UDP and web-ui on 943TCP), and are the defaults
* Nothing in logs points to the daemon failing (there's 8 daemons and they all start fine).
* going to https://10.0.1.86 gives me a ERR_CONNECTION_REFUSED
2) I tried connecting using the .ovpn profile connected. It seems like 1194 are failing to connect on the client side.
3) I was able to get the client connection working by forcing it to use TCP 443, with webui on 943 and no forwarding. This is obviously unideal since UDP is better. In this case everything works fine albiet through 443

I had a known working setup using 2.8.7 and as far as i can tell the configuration is the same. It appears that the "forwarding" functionality of TCP:443 isn't working, and something is up with 1194UDP for the actual VPN access.

let me know if there's any commands i can run to help diagnose this or if your team has noticed this same thing

{
"admin_ui.https.ip_address": "all",
"admin_ui.https.port": "943",
"aui.eula_version": "3",
"auth.ldap.0.name": "My LDAP servers",
"auth.ldap.0.ssl_verify": "internal",
"auth.ldap.0.timeout": "4",
"auth.ldap.0.use_ssl": "never",
"auth.ldap.0.user_exists_check": "true",
"auth.local.0.enable": "true",
"auth.module.type": "local",
"auth.pam.0.service": "openvpnas",
"auth.radius.0.acct_enable": "false",
"auth.radius.0.name": "My Radius servers",
"cs.admin_only": "false",
"cs.cws_proto_v2": "true",
"cs.cws_ui_offer.android": "false",
"cs.cws_ui_offer.autologin": "true",
"cs.cws_ui_offer.ios": "false",
"cs.cws_ui_offer.linux": "false",
"cs.cws_ui_offer.mac": "false",
"cs.cws_ui_offer.mac_v3": "false",
"cs.cws_ui_offer.server_locked": "false",
"cs.cws_ui_offer.user_locked": "true",
"cs.cws_ui_offer.win": "false",
"cs.cws_ui_offer.win_v3": "true",
"cs.https.ip_address": "all",
"cs.https.port": "943",
"cs.prof_sign_web": "true",
"cs.tls_version_min": "1.2",
"host.name": "redacted.com",
"sa.compression_warning_shown": "displayed",
"sa.initial_run_groups.0": "web_group",
"sa.initial_run_groups.1": "openvpn_group",
"upgrade.current_version": "2.11.3",
"upgrade.initial_version": "2.11.3",
"vpn.client.basic": "false",
"vpn.client.cipher": "AES-256-CBC",
"vpn.client.config_text": "",
"vpn.client.routing.inter_client": "true",
"vpn.client.routing.reroute_dns": "true",
"vpn.client.routing.reroute_gw": "false",
"vpn.client.routing.superuser_c2c_access": "false",
"vpn.daemon.0.client.netmask_bits": "26",
"vpn.daemon.0.client.network": "10.0.1.128",
"vpn.daemon.0.listen.ip_address": "all",
"vpn.daemon.0.listen.port": "443",
"vpn.daemon.0.listen.protocol": "tcp",
"vpn.daemon.0.server.ip_address": "all",
"vpn.general.osi_layer": "2",
"vpn.server.cipher": "AES-256-CBC",
"vpn.server.config_text": "",
"vpn.server.daemon.enable": "true",
"vpn.server.daemon.ovpndco": "false",
"vpn.server.daemon.protocols": "both",
"vpn.server.daemon.tcp.n_daemons": "4",
"vpn.server.daemon.tcp.port": "443",
"vpn.server.daemon.udp.n_daemons": "4",
"vpn.server.daemon.udp.port": "1194",
"vpn.server.data_ciphers": "",
"vpn.server.dhcp_option.disable_nbt": "false",
"vpn.server.dhcp_option.nbt": "1",
"vpn.server.duplicate_cn": "false",
"vpn.server.enable_cipher_fallback": "false",
"vpn.server.foreign_bridge": "",
"vpn.server.group_pool.0": "10.0.1.0/24",
"vpn.server.port_share.enable": "true",
"vpn.server.port_share.ip_address": "1.2.3.4",
"vpn.server.port_share.port": "1234",
"vpn.server.port_share.service": "admin+client",
"vpn.server.routing.gateway_access": "false",
"vpn.server.routing.private_access": "nat",
"vpn.server.routing.private_network.0": "10.0.1.0/24",
"vpn.server.tls_cc_security": "tls-auth",
"vpn.server.tls_version_min": "1.2",
"vpn.tls_refresh.interval": "60",
"xmlrpc.relay_level": "1"
}

alex.ong

Here's a simple sample when i used server on 10.0.1.86, and client on 10.0.1.200 trying to connect to same network
you can see the client is failing to do anything on 1194 (no logs on server side), and it maybe connects on 443TCP but server times it out.

Client:

2023-02-08 16:24:07 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). OpenVPN ignores --cipher for cipher negotiations. 
2023-02-08 16:24:07 Note: dev-type not tun, disabling data channel offload.
2023-02-08 16:24:07 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Feb  6 2023
2023-02-08 16:24:07 Windows version 10.0 (Windows 10 or greater), amd64 executable
2023-02-08 16:24:07 library versions: OpenSSL 3.0.7 1 Nov 2022, LZO 2.10
2023-02-08 16:24:07 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25343
2023-02-08 16:24:07 Need hold release from management interface, waiting...
2023-02-08 16:24:08 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:52289
2023-02-08 16:24:08 MANAGEMENT: CMD 'state on'
2023-02-08 16:24:08 MANAGEMENT: CMD 'log on all'
2023-02-08 16:24:08 MANAGEMENT: CMD 'echo on all'
2023-02-08 16:24:08 MANAGEMENT: CMD 'bytecount 5'
2023-02-08 16:24:08 MANAGEMENT: CMD 'state'
2023-02-08 16:24:08 MANAGEMENT: CMD 'hold off'
2023-02-08 16:24:08 MANAGEMENT: CMD 'hold release'
2023-02-08 16:24:08 TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:08 Socket Buffers: R=[65536->65536] S=[65536->65536]
2023-02-08 16:24:08 UDPv4 link local: (not bound)
2023-02-08 16:24:08 UDPv4 link remote: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:08 MANAGEMENT: >STATE:1675833848,WAIT,,,,,,
2023-02-08 16:24:12 Server poll timeout, restarting
2023-02-08 16:24:12 SIGUSR1[soft,server_poll] received, process restarting
2023-02-08 16:24:12 MANAGEMENT: >STATE:1675833852,RECONNECTING,server_poll,,,,,
2023-02-08 16:24:12 TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:12 Socket Buffers: R=[65536->65536] S=[65536->65536]
2023-02-08 16:24:12 UDPv4 link local: (not bound)
2023-02-08 16:24:12 UDPv4 link remote: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:12 MANAGEMENT: >STATE:1675833852,WAIT,,,,,,
2023-02-08 16:24:16 Server poll timeout, restarting
2023-02-08 16:24:16 SIGUSR1[soft,server_poll] received, process restarting
2023-02-08 16:24:16 MANAGEMENT: >STATE:1675833856,RECONNECTING,server_poll,,,,,
2023-02-08 16:24:16 TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.1.86:443
2023-02-08 16:24:16 Socket Buffers: R=[65536->65536] S=[65536->65536]
2023-02-08 16:24:16 Attempting to establish TCP connection with [AF_INET]10.0.1.86:443
2023-02-08 16:24:16 MANAGEMENT: >STATE:1675833856,TCP_CONNECT,,,,,,
2023-02-08 16:24:20 TCP: connect to [AF_INET]10.0.1.86:443 failed: Unknown error
2023-02-08 16:24:20 SIGUSR1[connection failed(soft),connection-failed] received, process restarting
2023-02-08 16:24:20 MANAGEMENT: >STATE:1675833860,RECONNECTING,connection-failed,,,,,
2023-02-08 16:24:20 Restart pause, 1 second(s)
2023-02-08 16:24:21 TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:21 Socket Buffers: R=[65536->65536] S=[65536->65536]
2023-02-08 16:24:21 UDPv4 link local: (not bound)
2023-02-08 16:24:21 UDPv4 link remote: [AF_INET]10.0.1.86:1194
2023-02-08 16:24:21 MANAGEMENT: >STATE:1675833861,WAIT,,,,,,
2023-02-08 16:24:23 SIGTERM[hard,] received, process exiting
2023-02-08 16:24:23 MANAGEMENT: >STATE:1675833863,EXITING,SIGTERM,,,,,

Server:

2023-02-08T05:24:47+0000 [stdout#info] [WEB] OUT: "2023-02-08T05:24:47+0000 [twisted.web.http.HTTPChannel#info] Timing out client: IPv4Address(type='TCP', host='10.0.1.200', port=52267)"
2023-02-08T05:24:47+0000 [stdout#info] [WEB] OUT: "2023-02-08T05:24:47+0000 [twisted.web.http.HTTPChannel#info] Timing out client: IPv4Address(type='TCP', host='10.0.1.200', port=52269)"
2023-02-08T05:24:47+0000 [stdout#info] [WEB] OUT: "2023-02-08T05:24:47+0000 [twisted.web.http.HTTPChannel#info] Timing out client: IPv4Address(type='TCP', host='10.0.1.200', port=52268)"
2023-02-08T05:24:47+0000 [stdout#info] [WEB] OUT: "2023-02-08T05:24:47+0000 [twisted.web.http.HTTPChannel#info] Timing out client: IPv4Address(type='TCP', host='10.0.1.200', port=52282)"
2023-02-08T05:24:47+0000 [stdout#info] [WEB] OUT: "2023-02-08T05:24:47+0000 [twisted.web.http.HTTPChannel#info] Timing out client: IPv4Address(type='TCP', host='10.0.1.200', port=52266)"

openvpn_inc

Hello alex.ong,

We tested 2.11.3 on Ubuntu 22.04 and can assure you it is capable of doing its job there and listen on UDP 1194 just fine.

However, there may be something in your particular situation that's somehow a problem. I notice your configuration is an existing one (or at least has a bunch of changes from the default). So I have to interpret "I set up a fresh VM" as meaning you did set up a new VM but copied configuration from an existing Access Server (or changed a bunch of settings). So the configuration was upgraded from some previous version (probably). That's something we also test for, but there may be a particular setting in your case that could be a problem. Especially since there are about 3 years worth of changes between your old version and new version.

I notice this in your configuration:
"vpn.server.daemon.tcp.n_daemons": "4",
"vpn.server.daemon.udp.n_daemons": "4",

If you don't actually have at least 4 CPU cores, the web interface will become unstable because of this. You could try setting this to 1 for both and see if things start behaving then. Your new VM might have a different amount of CPU cores than your old system. You can set it with these commands:
/usr/local/openvpn_as/scripts/sacli --key "vpn.server.daemon.tcp.n_daemons" --value "1" configput
/usr/local/openvpn_as/scripts/sacli --key "vpn.server.daemon.udp.n_daemons" --value "1" configput
/usr/local/openvpn_as/scripts/sacli start

I notice this in your configuration:
"vpn.daemon.0.client.network": "10.0.1.128",
"vpn.daemon.0.client.netmask_bits": "26",

So you've apparently configured things so the VPN clients are assigned IP addresses in the 10.0.1.128/26 range. But your server is at address 10.0.1.86. Is that a /24 range? That would mean your VPN clients are assigned IP addresses within that range which would be a conflict. VPN client subnet should be something unique and not part of your existing subnets. This could be a problem. The solution would be to give the VPN clients their own unique subnet like 172.16.55.0/24 or something. You can set that with these commands:
/usr/local/openvpn_as/scripts/sacli --key "vpn.daemon.0.client.network" --value "172.16.55.0" configput
/usr/local/openvpn_as/scripts/sacli --key "vpn.daemon.0.client.netmask_bits" --value "24" configput
/usr/local/openvpn_as/scripts/sacli start

If that doesn't help I would suggest to do a big easy debugging step first. I suggest you set up a new VM again but this time do not copy your old configuration into it. Set it up as a new Access Server and see if the web port redirection is working correctly now. If that works, then try copying your configuration into it again. Does it fail now? Then obviously it's something in the configuration. Then that's a starting point to work from. If however it already fails with a fresh configuration, it's obviously something in the Ubuntu 22 environment. In that case I would suggest to see if there is a firewall or HTTPS server software installed already and turning that off or removing it, to ensure it cannot interfere.

If a fresh install works you could consider then stopping Access Server, copying your certs.db and userprop.db, and starting Access Server. See if things continue to work then. Doing so would give you back your users and certificates. If it still works, you can copy other config values one by one and retest things until you hit the value that breaks things on your particular system.

If you want a faster response in debugging this you might try our support ticket system at https://openvpn.net/support

Good luck,
Johan

alex.ong

Hi Johan,

Firstly, thanks for your extremely prompt response!

I notice your configuration is an existing one (or at least has a bunch of changes from the default).

Yep! I used default then as part of debugging I slowly copied across settings from the other one. 80% sure the 443 -> 943 passthrough wasn't working to start with but i'll do it from scratch again to verify!

If you don't actually have at least 4 CPU cores

It's a VM with 4 cores but i can set it to 1.

vpn.daemon.0.client.network": "10.0.1.128",
"vpn.daemon.0.client.netmask_bits": "26",

Yep i copied this since it wasnt working with the default but imo this doesn't even do anything since:
1) 1194/443 is timing out (can't get an IP if it can't connect)
2) it's a bridged network / OSI layer 2 network anyway (so ip address handled by router on 10.0.1.1)
3) when I set the ports such that there isn't the forwarding, (443 for daemon, no webui passthrough), connecting via 443 TCP works.
Regardless i'll start from scratch.

I suggest you set up a new VM again but this time do not copy your old configuration into it. Set it up as a new Access Server and see if the web port redirection is working correctly now. If that works, then try copying your configuration into it again. Does it fail now? Then obviously it's something in the configuration. Then that's a starting point to work from. If however it already fails with a fresh configuration, it's obviously something in the Ubuntu 22 environment. In that case I would suggest to see if there is a firewall or HTTPS server software installed already and turning that off or removing it, to ensure it cannot interfere.

Brilliant, will do! When i ran nmap / lsof, nothing is using port 443 or 1194. There's no firewall on so definitely a headscratcher. I read that OpenVPN wont have anything showing up on 443 / 1194 because it uses iptables magic, so i assumed no entries was a good sign...

alex.ong

Alright, i've narrowed it down.
100% stock setup, website works correctly at https://10.0.1.86/ and https://10.0.1.86/admin. My apologies for the mistaken ACCUSATIONS :oops:

As soon as i run

cd /usr/local/openvpn_as/scripts/
sudo ./sacli --key "vpn.general.osi_layer" --value "2" ConfigPut
sudo ./sacli stop
sudo ./sacli start

it breaks the 443/943 forwarding so I have to use https://10.0.1.86:943 and https://10.0.1.86:943/admin

I've done further testing; my theory was that in the conversion from python 2.7 to 3.x, the forwarding stuff broke.
I couldnt install before 2.10.x on Ubuntu 22.04 (fair call there) so i installed a fresh 20.04.5 install and started from there:

| Ubuntu Server Version | OpenVPN Version          | https://10.0.1.86/ with osi_layer=2 |
|-----------------------|--------------------------|-------------------------------------|
| 22.04.1               | 2.11.3                   | fail                                |
|                       | 2.10.3                   | fail                                |
|                       | 2.10.x (earliest listed) | fail                                |
------------------------------------------------------------------------------------------
| 20.04.5               | 2.11.3                   | fail                                |
|                       | 2.10.3                   | fail                                |
|                       | 2.9.0                    | fail                                |
|                       | 2.8.8                    | pass                                |
| 20.04.1               | 2.7.5                    | pass                                |

Testing methodology
1. Pull ubuntu updates; run tzdata
2. Add repo
3. For each version:
i. install

apt install openvpn-as=2.8.x-asdfasdf

(set openvpn linux user's password if required for older versions)
ii. Confirm it works with https://10.0.1.86
iii.

./sacli --key "vpn.general.osi_layer" --value "2" ConfigPut
./sacli start

iv.

sudo apt purge openvpn-as

sudo rm-rf /usr/local/openvpn_as/

alex.ong

I raised a ticket with Support
TL;DR: Bridge mode is deprecated. I can make it work with routing so that's not too big an issue.

note that bridge mode still works in 2.11.3; simply set it to 1 daemon -> UDP.