Upgrade to OpenVPN 1.2.5 (iOS): DNS settings not applied

[ordex]

As I said in another thread: not yet, as we are pulling together other bug fixes. However this is our top priority at the moment, therefore we'll try to minimize any delay.

[risyer]

Sounds good, thanks ordex!

[oat_bondmen]

Sounds good, thanks ordex!

Does it sound good? Not to me it doesn't.

OpenVPN has pushed a release not fit for purpose, and instead of holding their hands up and admitting they have massively cocked up they instead put out a sham of a statement (written in bad English!) that only concerns itself with the change in API for accessing the iOS keychain (an API they were NOT forced to use in iOS11!).

Instead of banning people who are calling this out for the shambles it is, when are OpenVPN going to respond to the plethora of complaints and issues raised in a better way than a technically and grammatically inaccurate statement on your website?

[dazo]

@oat_bondmen: Your attitude is quite disturbing and disrespectful. You also conclude without any information about what has happened in the OpenVPN Connect iOS code base. You essentially lack information to support your claims. The updated OpenVPN Connect iOS app is required by Apple to move towards the new API because Apple is going to remove the old API. Apple have requested OpenVPN Inc to fix this.

This new API works completely different from the old one, it also has a much stricter security regime than the old one. So what the iOS app could do before we now need different techniques to achieve a similar user experience.

Yes, we acknowledge that quite some users experience issues now. And we have discovered another bug in the OpenVPN 3 Core library as well (with ill configured --tls-auth) On the other hand, our testing with external users also provided clear indications we fixed problematic configurations. So while you have issues now does not mean others have the same experience.

And we are working on fixing these issues. Currently, fixing this have our top priority. We rather want to spend time fixing this and release a new version than needing to react to false complaints on the forums and other channels.

[oat_bondmen]

@Dazo

I care not if you find my attitude 'quite disturbing' or 'disrespectful'. Respect is earned, not given.

Anyway, let's start with dissecting your post.

You also conclude without any information about what has happened in the OpenVPN Connect iOS code base. You essentially lack information to support your claims.

Of course we lack information; we are not your iOS engineering team. OpenVPN as a company should aim for a far higher transparency in responding to this release. Your post is the most insight I have read. Why did it take such prompting before the company posted this?

The updated OpenVPN Connect iOS app is required by Apple to move towards the new API because Apple is going to remove the old API. Apple have requested OpenVPN Inc to fix this.

A couple of points.

1. The API used is not being deprecated in iOS11. I've checked my Apple developer notes and see no reference to any deprecation of the currently utilised API for iOS11. I have no doubt it will be deprecated with iOS12, but that gives you many months to develop, test and resolve bugs found during development. No reasonable explanation has yet been provided as to why OpenVPN rushes this broken release out this week.

2. What date did Apple provide to you to issue an updated app against their 2014 (not a 'new' API as your company statement falsely claims) API? Was there a specific deadline you needed to meet with Apple causing you to release an incomplete and buggy application?

3. What regression tests were performed against 1.2.5 before release? There is basic functionality missing such as the inability to rename profiles and repeat 'Profile not enabled' errors when utilising multiple configuration profiles. How did basic issues such as these fail to be detected in QA?

Bottom line: Unless Apple forced you to release 1.2.5 this week (which, they didn't) there is no justification for pushing the release this week in its current broken state.

None.

This new API works completely different from the old one, it also has a much stricter security regime than the old one. So what the iOS app could do before we now need different techniques to achieve a similar user experience.

So? What's the relevance?

All this highlights is that your engineering team doesn't know the basics of major release numbers. You released this as a POINT RELEASE with limited release notes on the major invasive changes relating to Keychain handling. How in any situation do you think releasing an application with MAJOR changes warrants a simple point release update?

Yes, we acknowledge that quite some users experience issues now. And we have discovered another bug in the OpenVPN 3 Core library as well (with ill configured --tls-auth) On the other hand, our testing with external users also provided clear indications we fixed problematic configurations.

Sorry, what is this illiterate nonsense?

What does 'our testing with external users also provided clear indications we fixed problematic configurations' even mean? What problematic configurations do you believe have been resolved with 1.2.5?

So while you have issues now does not mean others have the same experience.

Again, this is jibbering nonsense. You have widely reported issues affecting the 1.2.5 all of which have been introduced as new bugs from the previous version.

And we are working on fixing these issues. Currently, fixing this have our top priority. We rather want to spend time fixing this and release a new version than

than....adequately beta test our broken product before releasing it to an unsuspecting public with a minor release version and incomplete release notes

FIFY.

needing to react to false complaints on the forums and other channels.

There are no 'false complaints' on this forum. For you to suggest this is offensive to those who are better informed. If you're a developer, you should be ashamed of this release and the response to it from your company.

[jbkf1003]

I'm a paying corporate user, and I have users who are affected. I'm upset as well, spouting off on a forum doesn't accomplish anything though. Microsoft and Apple have had their fair share of bad patches (and OSes for that matter, Windows ME anyone). Lets just work on getting it fixed.

[flashcrash]

another bug, unless you have redirect-gateway def1 on client side, DNS settings will not be received by client (pushed from server)...

Confirmed. Using the 1.1.1 client, DNS settings are successfully pushed from the server and applied. Using the 1.2.5 client they're not applied. Tested with iOS 11.2. FYI: The routes are good. If the remote system names are all substituted with their respective IP addresses instead, they're accessible. But clearly that's a pretty expensive and time-consuming workaround to apply for multiple clients and multiple remote system names.

[dazo]

@Dazo

I care not if you find my attitude 'quite disturbing' or 'disrespectful'. Respect is earned, not given.

And this is why I am not going to continue discussing this further with you. You have not earned enough credits to have our respect through your clueless responses. You talk and behave like you know how things are done, without any background information or knowledge of the various processes before this release. What you contribute with is plain noise. And I've already wasted enough time on this nonsense. Continuing this nonsense discussion will not fix the current issues. EOD.

[dazo]

another bug, unless you have redirect-gateway def1 on client side, DNS settings will not be received by client (pushed from server)...

Confirmed. Using the 1.1.1 client, DNS settings are successfully pushed from the server and applied. Using the 1.2.5 client they're not applied. Tested with iOS 11.2. FYI: The routes are good. If the remote system names are all substituted with their respective IP addresses instead, they're accessible. But clearly that's a pretty expensive and time-consuming workaround to apply for multiple clients and multiple remote system names.

Thanks for your report and testing! Yes, we believe we've identified the problem to a detail in the newer VPN API we now use. For some odd reasons, it can look like it need an explicit route to the DNS server. We're investigating and testing this further and will release an updated version ASAP.

Could you try to add this to the server config ... or a --client-config-dir file?

View OriginalPossible workaround

push "route $DNS_SERVER_IP 255.255.255.255"

Replace $DNS_SERVER_IP with the IP address of the DNS server you use.

[e-]

@oat_bondmen: Your attitude is quite disturbing and disrespectful. You also conclude without any information about what has happened in the OpenVPN Connect iOS code base. You essentially lack information to support your claims. The updated OpenVPN Connect iOS app is required by Apple to move towards the new API because Apple is going to remove the old API. Apple have requested OpenVPN Inc to fix this.

This new API works completely different from the old one, it also has a much stricter security regime than the old one. So what the iOS app could do before we now need different techniques to achieve a similar user experience.

Yes, we acknowledge that quite some users experience issues now. And we have discovered another bug in the OpenVPN 3 Core library as well (with ill configured --tls-auth) On the other hand, our testing with external users also provided clear indications we fixed problematic configurations. So while you have issues now does not mean others have the same experience.

And we are working on fixing these issues. Currently, fixing this have our top priority. We rather want to spend time fixing this and release a new version than needing to react to false complaints on the forums and other channels.

Hello dazo,

I appreciate the efforts in fixing the massive problems this release has caused, but I need to point out some things.

Your official blog post, ordex's replies, your reply and the state of the app do not match. The update's changelog states what sounds like minor changes/bug fixes between 1.2.4 and 1.2.5, but the blog post states that this version is a major overhaul of the code. ordex has been confusingly debating versions with users on this forum, because they read the misleading changelog saying 1.2.4 to 1.2.5, but in fact they had version 1.1.1 before the update. Why does the official changelog have internal beta information?

Why is a major overhaul version released as a minor change? Why did you not warn users that such significant changes were coming?

This affects corporate users who require the app for work, people under oppressive regimes who can't even access this very forum without a VPN and probably more types of users in-between. Currently this app is unusable for many people and this massive problem could have been easily avoided with better communication.

And the worst part: I bet many people will from now on, grow a very bad habit because of this mess. They will refuse to update in fear of getting broken again. Many have even stated so on this forum.

Please do a very thorough review on your organization's transparency policy, because this is a train wreck of a situation.

[oat_bondmen]

@Dazo

I care not if you find my attitude 'quite disturbing' or 'disrespectful'. Respect is earned, not given.

And this is why I am not going to continue discussing this further with you.

You are unable to continue this discussion because I have identified a litany of incompetence both prior to this release and subsequently.

You have not earned enough credits to have our respect through your clueless responses. You talk and behave like you know how things are done

I talk and behave as such because, well, er, I do know how things (should be!) done. That you feel able to speak condescendingly to your better, despite being part of this shit show that has simply served to demonstrating that OpenVPN Inc is an amateur at both planning, developing and QAing a major release.

You didn't even think to increase the version number to demonstrate the major changes; instead you went for a minor release! I'm sorry, but you can lecture _no one_ through this gross display of incompetence.

without any background information or knowledge of the various processes before this release.

So inform us!

Don't hide behind the drivel of a statement issued on your website which is woefully incompetent and offers no insight into the decision making process nor does it answer questions I and others have raised about the process leading to this broken release.

Continuing this nonsense discussion will not fix the current issues. EOD.

On the contrary, your reply has simply confirmed that you released this version despite no pressure from Apple to do so before it was ready. You have further confirmed that your QA process is not fit for purpose, nor do you know how to engage with unhappy customers.

You - and your organisation in general - have a _lot_ to learn both about release processes and how to react when things go badly.

[dazo]

Your official blog post, ordex's replies, your reply and the state of the app do not match. The update's changelog states what sounds like minor changes/bug fixes between 1.2.4 and 1.2.5, but the blog post states that this version is a major overhaul of the code. ordex has been confusingly debating versions with users on this forum, because they read the misleading changelog saying 1.2.4 to 1.2.5, but in fact they had version 1.1.1 before the update. Why does the official changelog have internal beta information?

We don't intend to be confusing. The 1.2.4 was the last beta-release which we made available to selected external users through testflight. These users where users who had severe issues using the last stable release (1.1.1). We fixed several issues there before the final 1.2.5 release which got public to everyone. That's is basically what has happened.

Why is a major overhaul version released as a minor change? Why did you not warn users that such significant changes were coming?

I'll admit I don't know much about how things are tagged in the Appstore when we push out updates (that's done by a few privileged people in the company). But the version number itself can be misunderstood; which I think is why several people see the change from 1.1.1 to 1.2.5 as a minor update. But for us moving to 1.1 to 1.2 is a major change. And this follows the same regime we've used in the OpenVPN 2.x releases as well. OpenVPN 2 indicates the generation, OpenVPN 2.4 is the major release and 2.4.4 is the minor release.

In hindsight, we should have put up a more massive notification that this is a major upgrade. But that is not something we can change currently, and we will just need to resolve the situation as it is now.

This affects corporate users who require the app for work, people under oppressive regimes who can't even access this very forum without a VPN and probably more types of users in-between. Currently this app is unusable for many people and this massive problem could have been easily avoided with better communication.

This is truly unfortunate and we are truly sorry for this breakage. On the other hand, we have also received reports from users that this release makes things better for them; those users are happy. So things aren't as black-or-white as this discussion thread may give an impression of.

And the worst part: I bet many people will from now on, grow a very bad habit because of this mess. They will refuse to update in fear of getting broken again. Many have even stated so on this forum.

Well, this is both good and bad. It is good to test an upgrade when you have time to investigate if things breaks. It is bad if you're so swamped you never have time to test an upgrade and therefore refuse to upgrade. But this is not a situation which is unique to OpenVPN Connect on iOS. This is a generic issue. I also don't update apps I depend on in my daily life if I know I don't have time to debug it. But I do it asap as I have time. And I don't think this is uncommon practice at all.

Please do a very thorough review on your organization's transparency policy, because this is a train wreck of a situation.

We do indeed try to be as transparent as possible. We have nothing to hide in our updates, so being transparent is what we do strive for. But there's also a fine line between being too detailed and too dense. We need to strike the fine balance to ensure non-technical users gets an understanding why an update is important as well; while carrying enough information to the more tech savy audience. But rest assure, we do not intend to keep our users in the dark. Which is why we do spend time responding to the forums and other channels. We want to be an open company with a thriving and supportive community involved. I'm not saying we're perfect and doing everything optimally, but we do want to learn from our mishaps and play along with our community of users and developers.

[e-]

Well, this is both good and bad. It is good to test an upgrade when you have time to investigate if things breaks. It is bad if you're so swamped you never have time to test an upgrade and therefore refuse to upgrade. But this is not a situation which is unique to OpenVPN Connect on iOS. This is a generic issue. I also don't update apps I depend on in my daily life if I know I don't have time to debug it. But I do it asap as I have time. And I don't think this is uncommon practice at all.

That is indeed common and I usually do the same. I am affected by the update, but fortunately I have alternatives. The point that bothers me more than I'd like to admit is how misleading the update notes were. Having used OpenVPN GUI for a while now (and applying nearly all updates for it as they were made available), then reading the seemingly harmless update notes, I incorrectly judged that the update was safe.

The key difference is that iOS has no rollback options, so things like these cause problems most users cannot deal with. I am technically capable to do the manual rollback linked on this forum, but lack time to do it now (might do over the weekend). People close to me are complaining about it and asking for a solution, so I may need to spend the weekend fixing their devices.

To reiterate: I understand the need to move forward and the challenges that often presents. But communication in this case was very poor all around. Had I known that such a massive change was coming, I'd have taken a different approach to this update.

I wasn't even aware that you guys use Testflight before this happened, and I'd be willing to help if needed.

Thank you for taking the time to answer, I'll leave you to work on the issues now.

[ordex]

View OriginalPossible workaround

push "route $DNS_SERVER_IP 255.255.255.255"

Replace $DNS_SERVER_IP with the IP address of the DNS server you use.

Unfortunately this is not enough. The only workaround we found so far for this is to enable gateway-redirect and temporary pass all the traffic through the server.

[dazo]

Unfortunately this is not enough. The only workaround we found so far for this is to enable gateway-redirect and temporary pass all the traffic through the server.

Ahh ... sorry about the misleading noise!

[flashcrash]

Could you try to add this to the server config ... or a --client-config-dir file: push "route $DNS_SERVER_IP 255.255.255.255"

Tried it in the server config. Doesn't seem to make any difference unfortunately.

[tos42]

Hi,

same problem here. The workaround with redirect-gateway works, although it can't be a permanent solution as all internet traffic from the device goes through VPN connection...

It seems to be related to the Apple Network Extensions migration, do you have any clues on how to permanently resolve this behaviour ?

--

[manchik]

Hi ALL,

Same here. Does not seem to affect iOS 9.x versions but 11.x does. Not sure about 10.x though.

[ordex]

Hi,

same problem here. The workaround with redirect-gateway works, although it can't be a permanent solution as all internet traffic from the device goes through VPN connection...

It seems to be related to the Apple Network Extensions migration, do you have any clues on how to permanently resolve this behaviour ?

--

Hi there,
this answer was given some posts above.
We already have a solution and it has been confirmed to be working by our beta testers.

The next release will be unaffected

[tos42]

Hi,

same problem here. The workaround with redirect-gateway works, although it can't be a permanent solution as all internet traffic from the device goes through VPN connection...

It seems to be related to the Apple Network Extensions migration, do you have any clues on how to permanently resolve this behaviour ?

--

Hi there,
this answer was given some posts above.
We already have a solution and it has been confirmed to be working by our beta testers.

The next release will be unaffected

Thanks ordex, do you know approximatively how much time will it take to release this unaffected new version ?