Client certificates revoked after upgrade to 2.11

Business solution to host your own OpenVPN server with web management interface and bundled clients.
Post Reply
chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Client certificates revoked after upgrade to 2.11

Post by chort1 » Wed Aug 24, 2022 8:01 am

After our last server patching (RHEL8), Access server was upgraded to version 2.11. After this, several users are getting error message "Client certificate has been revoked" when trying to log in via the client.

Checking the admin panel, I can see that 70-80% of the user profiles are gone. If the users log on to the web GUI and create a new profile, they can log in with the client again, using the new profile, though that's far from ideal.

Is this a known issue? It coincides quite heavily with the upgrade, but if there's any type of troubleshooting that might prove otherwise, I'd very much like to investigate.

Edit: I can add that this is a standalone server, but uses MariaDB (running locally) as backend.

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1333
Joined: Tue Feb 16, 2021 10:41 am

Re: Client certificates revoked after upgrade to 2.11

Post by openvpn_inc » Wed Aug 24, 2022 8:09 am

Hello chort1,

Sounds like something in your environment went wrong. This is definitely not a known issue, it shouldn't happen. But there may be something in the database backend going wrong, but without more details I simply don't know. Since you rely on an external database backend I hope you have backups there. If so you could consider a rollback and restoring an older copy of the database, so that you can then have everyone connect again without issues, and then set up a test environment to run the upgrade there and see if any issues show up there and discuss them with our technical support guys on our support ticket system.

Kind regards,
Johan
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Re: Client certificates revoked after upgrade to 2.11

Post by chort1 » Wed Aug 24, 2022 9:39 am

Hi Johan, and thanks for the quick reply.

Am I not correct in assuming that MySQL/MariaDB backend is a supported configuration? It's been working fine up until now, and it seems strange that a faulty db backend would remove only some of the users... I guess we can dump the as_certs database and restore it, though users can fix the issue by creating new profiles for themselves, if this is a one time occurrence. We have a patch window for another (similar) Access server installation tonight, so we'll see how that goes.

chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Re: Client certificates revoked after upgrade to 2.11

Post by chort1 » Wed Aug 24, 2022 11:15 am

Some more information, after a bit of troubleshooting

It seems that after this incident, some users have had the serial number on their profiles changed.

Example:
Yesterday Bob's profile was number 108, and today it was 44.
Yesterday Alice's profile was number 44, and today it was gone
When Alice tries to sign on, it fails, and we get a log entry that says:

Code: Select all

username-only match fail, client username=\alice\', DB username=\'bob\'

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1333
Joined: Tue Feb 16, 2021 10:41 am

Re: Client certificates revoked after upgrade to 2.11

Post by openvpn_inc » Wed Aug 24, 2022 12:21 pm

Hi,

What benefit do you see from using mariadb over sqlite3? I would not think there is any. It gives you another moving part which can break.

What was the version you upgraded from, was it 2.8.x or earlier? We had a schema change in 2.9 (same schema through 2.11).

Support ticket is a good idea. We do provide support for free tier users as time permits; not limited to paying customers.

regards, rob0
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Re: Client certificates revoked after upgrade to 2.11

Post by chort1 » Wed Aug 24, 2022 12:38 pm

Hi rob0

The choice of mariadb was a request from operations, who were more comfortable maintaining and troubleshooting this than sqlite.

We upgraded via yum from 2.10.3_c47a813c-1 to 2.11.0_794ab41d-1 on RHEL8.6

After some further troubleshooting it looks like all profiles over a certain limit (41 being the highest) were removed. The renumbering was likely a consquence of users requesting new profiles and being assigned the newly freed-up serial numbers

In other words, it looks like the database was somehow reverted to an earlier stage. Could be related to the older sqlite-files still being present in /usr/local/etc/db ...?

Will consider a support ticket. We have a subscription, by the way.

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1333
Joined: Tue Feb 16, 2021 10:41 am

Re: Client certificates revoked after upgrade to 2.11

Post by openvpn_inc » Wed Aug 24, 2022 12:57 pm

Hi,

If as.conf points to mysql, the sqlite3 databases are not consulted.

As for "maintaining" the sqlite3 databases, Access Server does that without human help. Best to think of sqlite3 as a file type, rather than as a RDBMS. And troubleshooting with a command line tool, sqlite3(1), can be done just as with mariadb/mysql(1). (Granted, there are a few quirks of mysql's SQL implementation, which don't apply with sqlite3, but for the most part it is very similar, and querying the databases can't break anything.)

2.10 to 2.11 would use the same database schema. So I am puzzled here. At what point was it migrated to mysql?

If you do open a ticket let me know. I'm not in Support, but I will look in when I can.

regards, rob0
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Re: Client certificates revoked after upgrade to 2.11

Post by chort1 » Wed Aug 24, 2022 1:11 pm

I see your point about sqlite, but when you run a cluster, mysql is a requirement, so I imagined it would still be a supported configuration for a standalone...?

I see that in /usr/local/openvpn_as/etc/db/backup there are automatically created folders whose dates coincide with patch times, including one for yesterday. When I check the certs.db files in them, they seem unchanged from one folder to the next.

The migration to sql was done in April, and the certs.db-files contain only entries dated back then. 6 entries, as opposed to 41 that were left after upgrading the previous server.

If I make a ticket, I'll let you know. I think we'll see how patching the remaining server turns out tonight

chort1
OpenVPN User
Posts: 27
Joined: Tue Mar 01, 2022 12:24 pm

Re: Client certificates revoked after upgrade to 2.11

Post by chort1 » Thu Aug 25, 2022 6:35 am

Hi again, the problem has been resolved, and it was NOT an issue with the upgrade itself

At some point we moved /var/lib/mysql to a new partition and forgot to remove the old fstab entry, so we were left with two entries for the same path. After a reboot, the path was mounted on the old volume, containing SQL data from before the move.

Thank you for all replies, and sorry for wasting your time

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1333
Joined: Tue Feb 16, 2021 10:41 am

Re: Client certificates revoked after upgrade to 2.11

Post by openvpn_inc » Thu Aug 25, 2022 1:27 pm

Hi chort1,

So my initial guess about it being a database backend issue turned out to be true, nice. Glad you figured it out.

Thanks for reporting back.

Have a nice day,
Johan
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

Post Reply