upgrade 2.9 issue

[frank3427]

I upgraded my 2.8.8 installation to 2.9

after the upgrade I was not able to get the web interface. I did not find running process as in 2.8.8

2.8.8
root@vpn:~# ss -tupln | grep openvpn
udp UNCONN 0 0 10.108.24.21:1194 0.0.0.0:* users:(("openvpn-openssl",pid=1487,fd=5))
tcp LISTEN 0 32 10.108.24.21:443 0.0.0.0:* users:(("openvpn-openssl",pid=1475,fd=5))
root@vpnc:~#

2.9.0
root@vpn:~# ss -tupln | grep openvpn

I rollback to 2.8.8 an everything comes back

[chilinux]

OpenVPN Access Server 2.9.0 has several major changes including being modified to work in Python v3. It is possible not all the options/features have been fully tested for regression issues yet.

As long as the rollback to 2.8.8 is working for you, it is probably best to stay with that.

If you open a ticket with support, the should eventually be able to look into what is causing the issue.

If you want to troubleshoot the problem yourself, attempt accessing the admin web server on port 943 instead of 443. Also, take a look at the server logs from a system shell.

[openvpn_inc]

Hello frank,

You should contact us on the support ticket system at https://openvpn.net/support

We can then gather some information on your exact situation and where it went wrong, and provide steps to correct the problem. We are very interested to learn details about the problem you have experienced. If we can get the details of what went wrong, we can work on solving it. However, please do not post such details on this public forum. Please use our support ticket system, as that is secure.

Kind regards,
Johan

[beerfloat]

Well this is not great. You pushed 2.9.1 through automatic repositories like (in my case) yum, and when installed users can't connect anymore.
I find out connections only start working again after they redownload profiles.

This is not what I expect from commercial software.

[openvpn_inc]

Hello beerfloat,

We do the absolute maximum effort to ensure compatibility with all previous versions of OpenVPN Access Server, even going back to versions of 10 years ago. We do extremely extensive tests covering well over a thousand cases. But it is still possible that we missed some particular case. We request that you contact us on our support ticket system so that we can get details of your configuration, so that we can either give you the commands to resolve your problem, or if it is a bug, that we solve it on our end in the next release.

We would love to investigate further regarding the issue you just reported. The support ticket system at https://openvpn.net/support is the right place to securely send us some log files and other information we might need to find out what happened and solve it if possible.

Kind regards,
Johan

[chilinux]

I'm not sure it is completely to OpenVPN AS' advantage to claim the absolute maximum effort is already being done. It makes it sound like there is no room for improvement or that OpenVPN AS is not willing discuss the possibilities for improvement.

I would agree that OpenVPN AS has done an amazing job at testing the majority of the product. But the degree to which it has always ensured compatibility going back over 10 years has not always been true. If you go back far, you could enable Layer 2 / TAP VPN but the upgrade would no longer honor the configuration. It is simply not part of the core product and overly complicated it but removing it also impacts behavior. More recently, post_auth script users may have needed to update their scripts twice over the last year. Once when the python LDAP module was changed and a second when the product moved to python v3.

More importantly, for the most part, OpenVPN AS still take a very hands off approach to the upgrade path. It is left to the customer to track the release notes and perform upgrades external to the OpenVPN AS web portal. For someone that can act as an active system administrator, that is fine. For a novice that is expecting more of an appliance that takes care of itself, this might come as a disappointment.

Here are some thing I think might help (or possibly not) improve the experience for novice users:

(1) Include daily update checks on the Status Overview page. Make it easy for the admin to tell from the web portal when the OS or OpenVPN AS has an upgrade available.

(2) Include the option for update check emails to be sent to an administrator email address.

(3) Build an additional package that can perform checks for known issues that might impact upgrades and issue a warning. For example, an openvpn-as-upchk package for 2.8.8 may issue a warning to users with post_auth enabled that they may need to modify their script to upgrade.

(4) Make being able to schedule automated upgrades part of the web portal. The interface should make it easy for OS security updates to be applied and if there is no pending warning for upgrading to also perform upgrades of OpenVPN AS itself.

(5) Automate client side health checks and rollback for upgrades. It should be possible for the service that starts up OpenVPN AS for the first time after the upgrade to confirm the expected processes start without error and that the expected network ports are listen to. If the upgrade doesn't pass an automated upgrade inspection, it should revert the upgrade and notify the administrator via the Status Overview page and email.

(6) Add better telemetry/diagnostic collection tools. Other products I deal with include a script for gathering logs and other details about a system for the customer to run whenever unexpected behavior takes place. This helps expedite support.

(7) Have more than one concurrent release repository. Some other products have more than one release such as a "stable" and a "feature" release. In some cases this goes as far as being three different releases such as stable, beta and canary. The OpenVPN AS product goes straight from in-house testing to stable. There appears to be no option for customers to be part of an early-release group.

(8) Please be more transparent about known issues. I reported something back in February which was finally acknowledged by support a month later. However, it is still not publicly acknowledged at all. That leaves me wondering what other things OpenVPN AS support knows doesn't work as documented but no customer facing information is provided to help them plan accordingly.

[openvpn_inc]

Hello chilinux,

Your feedback is noted. Many of the items are already on our roadmap, however.

Just a note on Layer 2 - that still exists even now in AS 2.9. It's just hidden by default on a fresh install because we deprecated this feature. It's still present but due to it being deprecated we do not provide support for it anymore.

Kind regards,
Johan

[chilinux]

Hello chilinux,

Your feedback is noted. Many of the items are already on our roadmap, however.

Is the roadmap available to customers anyplace?

It would be nice to have access to that and be able to see some sort of status or ETA for roadmap items. As it is, I can't even get any form of meaningful status update on the bug I filed.

Just a note on Layer 2 - that still exists even now in AS 2.9. It's just hidden by default on a fresh install because we deprecated this feature. It's still present but due to it being deprecated we do not provide support for it anymore.

I'm glad it is hidden. I was not asking for it to ever come back, I was just pointing out that things change over time (and with good reason). I was only making an example of it to lead up to my point that OpenVPN AS may benefit from taking a more active role on the upgrade path.

[openvpn_inc]

Hello chilinux,

No, our roadmap is not published at this time, sorry.

If you let me know the ticket number of the ticket you sent in, I can personally review your case and provide a response in that ticket.

Kind regards,
Johan

[frank3427]

to day I tried upgrading to 2.9.2 from 2.8.8 it still fails to start. I know that in 2.9.0 release notes warning about Post_auth possible issue.
I have opened a ticket # 387056

here is what I was getting in the log, once again I was able to downgrade to 2.8.8 to get it working again, does anyone know where the post_auth script is located on the file sysytem?

2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 904'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 905'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 906'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 907'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 908'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] ASWebSite (TLS) starting on 909'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [-] set uid/gid 1001/1001'
2021-07-24T13:44:18+0000 [stdout#info] [WEB] OUT: '2021-07-24T13:44:18+0000 [stdout#info] Web server running as UID 1001'
2021-07-24T13:44:18+0000 [stdout#info] PROC SET /proc/sys/net/ipv4/ip_forward : b'1' -> 1
2021-07-24T13:44:18+0000 [stdout#info] PROC SET /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal : b'1' -> 1
2021-07-24T13:44:19+0000 [stdout#info] OpenVPNDataDir: using shared dir: '/run/openvpn_as/memstats'
2021-07-24T13:44:19+0000 [stdout#info] subscription: enforcement_order is not set. Will disconnect newest subscription clients
2021-07-24T13:44:19+0000 [stdout#info] Server Agent initialization status:
2021-07-24T13:44:19+0000 [stdout#info] {
2021-07-24T13:44:19+0000 [stdout#info] "errors": {
2021-07-24T13:44:19+0000 [stdout#info] "auth": [
2021-07-24T13:44:19+0000 [stdout#info] [
2021-07-24T13:44:19+0000 [stdout#info] "error",
2021-07-24T13:44:19+0000 [stdout#info] "error loading post_auth script: Missing parentheses in call to 'print'. Did you mean print(\"********** POST_AUTH\", authcred, attributes, authret, info)? (<string>, line 165): svc/svc:675,sagent/authsvc:114,sagent/authsvc:103 (SyntaxError)"
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] ],
2021-07-24T13:44:19+0000 [stdout#info] "client_query": [
2021-07-24T13:44:19+0000 [stdout#info] [
2021-07-24T13:44:19+0000 [stdout#info] "error",
2021-07-24T13:44:19+0000 [stdout#info] "service failed to start due to unresolved dependencies: {'auth'}"
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] ],
2021-07-24T13:44:19+0000 [stdout#info] "crl": [
2021-07-24T13:44:19+0000 [stdout#info] [
2021-07-24T13:44:19+0000 [stdout#info] "error",
2021-07-24T13:44:19+0000 [stdout#info] "service failed to start due to unresolved dependencies: {'auth'}"
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] ],
2021-07-24T13:44:19+0000 [stdout#info] "openvpn_0": [
2021-07-24T13:44:19+0000 [stdout#info] [
2021-07-24T13:44:19+0000 [stdout#info] "error",
2021-07-24T13:44:19+0000 [stdout#info] "service failed to start due to unresolved dependencies: {'auth'}"
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] ],
2021-07-24T13:44:19+0000 [stdout#info] "openvpn_1": [
2021-07-24T13:44:19+0000 [stdout#info] [
2021-07-24T13:44:19+0000 [stdout#info] "error",
2021-07-24T13:44:19+0000 [stdout#info] "service failed to start due to unresolved dependencies: {'auth'}"
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] ]
2021-07-24T13:44:19+0000 [stdout#info] },
2021-07-24T13:44:19+0000 [stdout#info] "last_restarted": "Sat Jul 24 13:44:17 2021",
2021-07-24T13:44:19+0000 [stdout#info] "service_status": {
2021-07-24T13:44:19+0000 [stdout#info] "api": "started",
2021-07-24T13:44:19+0000 [stdout#info] "auth": "off. Error: [Error: error loading post_auth script: Missing parentheses in call to 'print'. Did you mean print(\"********** POST_AUTH\", authcred, attributes, authret, info)? (<string>, line 165): svc/svc:675,sagent/authsvc:114,sagent/authsvc:103 (SyntaxError).]",
2021-07-24T13:44:19+0000 [stdout#info] "bridge": "started",
2021-07-24T13:44:19+0000 [stdout#info] "client_query": "off. Error: [Error: service failed to start due to unresolved dependencies: {'auth'}.]",
2021-07-24T13:44:19+0000 [stdout#info] "crl": "off. Error: [Error: service failed to start due to unresolved dependencies: {'auth'}.]",
2021-07-24T13:44:19+0000 [stdout#info] "daemon_pre": "started",
2021-07-24T13:44:19+0000 [stdout#info] "db_push": "started",
2021-07-24T13:44:19+0000 [stdout#info] "ip6tables_live": "started",
2021-07-24T13:44:19+0000 [stdout#info] "ip6tables_openvpn": "started",
2021-07-24T13:44:19+0000 [stdout#info] "iptables_live": "started",
2021-07-24T13:44:19+0000 [stdout#info] "iptables_openvpn": "started",
2021-07-24T13:44:19+0000 [stdout#info] "iptables_web": "started",
2021-07-24T13:44:19+0000 [stdout#info] "log": "started",
2021-07-24T13:44:19+0000 [stdout#info] "openvpn_0": "off. Error: [Error: service failed to start due to unresolved dependencies: {'auth'}.]",
2021-07-24T13:44:19+0000 [stdout#info] "openvpn_1": "off. Error: [Error: service failed to start due to unresolved dependencies: {'auth'}.]",
2021-07-24T13:44:19+0000 [stdout#info] "subscription": "started",
2021-07-24T13:44:19+0000 [stdout#info] "user": "started",
2021-07-24T13:44:19+0000 [stdout#info] "web": "started"
2021-07-24T13:44:19+0000 [stdout#info] }
2021-07-24T13:44:19+0000 [stdout#info] }
2021-07-24T13:44:19+0000 [stdout#info] Server Agent started
2021-07-24T13:44:48+0000 [-] DNSDatagramProtocol starting on 12475
2021-07-24T13:44:48+0000 [-] Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7fc5af97ce20>
2021-07-24T13:44:48+0000 [-] (UDP Port 12475 Closed)
2021-07-24T13:44:48+0000 [-] Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7fc5af97ce20>
2021-07-24T13:44:49+0000 [stdout#info] License Info {'concurrent_connections': 10, 'apc': False}
2021-07-24T13:45:48+0000 [-] DNSDatagramProtocol starting on 23291
2021-07-24T13:45:48+0000 [-] Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7fc5af985160>
2021-07-24T13:45:48+0000 [-] (UDP Port 23291 Closed)
2021-07-24T13:45:48+0000 [-] Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x7fc5af985160>

[chilinux]

2021-07-24T13:44:19+0000 [stdout#info] "error loading post_auth script: Missing parentheses in call to 'print'. Did you mean print(\"********** POST_AUTH\", authcred, attributes, authret, info)? (<string>, line 165): svc/svc:675,sagent/authsvc:114,sagent/authsvc:103 (SyntaxError)"

This is the part to focus on.

Python v2 allowed print to be a keyword instead of a function. As such this is valid in v2 but not v3:

Code: Select all

print "Hello World"

Instead, this is valid in both Python v2.7 and v3:

Code: Select all

print("Hello World")

This is just one issue with using your existing post_auth script with 2.9.x, there may be others.

Some of these can be fixed using Python's automated "2to3" code translation as explained here:
https://docs.python.org/3/library/2to3.html

This however does not always fix everything and sometimes additional changes by hand need to be made.

Until the post_auth script is updated to follow the requirements of Python v3, you will have to remain on 2.8.8 since that is still Python v2 based.

If you are willing to share your post_auth script, we can make suggestions on how to modify it.

If the post_auth script is considered confidential then you should provide the script to OpenVPN AS support and they can try to provide guidance on modifying it to Python v3. Given the amount of Python v2 code they must have just modified to make AS v2.9 a reality, I can imagine they have some of the developers that have gotten really good at spotting v2 to v3 changes that need to be made.

[frank3427]

I was able to resolve the issue her are my notes, my installation is on Ubuntu 20.04. this server start out on 16.04, upgrade to 18.04 then to 20.04 over time
##############
in the new version:

This release now runs on Python3 (previously Python2). The installation process will install necessary files.

Post_auth scripts may need to be updated for Python3 compatibility

we will need to update the following script on all server by downloading and installing the script on each server

as root user
cd /usr/local/openvpn_as/scripts
wget http://swupdate.openvpn.net/scripts/pos ... hecking.py

install the script
./sacli -k auth.module.post_auth_script --value_file=/usr/local/openvpn_as/scripts/post_auth_mac_address_checking.py ConfigPut


need to remove the apt-mark hold openvpn-as

apt-mark unhold openvpn-as
sudo apt update && sudo apt upgrade -y
sudo reboot is recommended