Involvement of FOX-IT in OpenVPN

palatinux

Dear OpenVPN users,

We've noticed that the security company FOX-IT is responsible for a lot of new code in the latest OpenVPN versions, which is nice because this has a positive effect on it's security.

But, as a Dutch security company, we are also familiar with the FOX-IT history. FOX-IT does not only provide security solutions, but they also also provide spyware, deep-packet-inspection and lawful interception solutions to goverments, like their product "FoxReplay" for example:

http://wikileaks.org/spyfiles/list/company-name/foxit.html

Can the OpenVPN developers guarantee us that every bit of code FOX-IT provides is carefully audited for backdoors? I assume we don't want to have a OpenBSD IPSEC/FBI scandal on this nice VPN product. Just imagen how much of their spy products they could sell when they have created a backdoor into OpenVPN.

We also like to inform you that PolarSSL in mainted by another former FOX-IT member Paul Bakker.

Note: We don't have the intention to start a flaming-war here, but this is something the OpenVPN users and developers should be aware of.

The Fortress Linux Security Team,
http://www.fortresslinux.org

maikcat

interesting stuff....

which version of openvpn has their code?

waiting a comment from openvpn developers.

Michael.

palatinux

We saw the first signs in OpenVPN 2.2.2, the latest 2.3 Alpha is quite loaded with FOX-IT work. Especially the PolarSSL implementation, PNRG function and the connection establisiments.

dazo

First of all, thank you very much for your critical question. This is surely appreciated!

When it comes to the patches which where added, it is really difficult to 100% guarantee that Fox-IT have not introduced any nasty stuff. These Fox-IT guys are really clever people. But also after having met them in real life during FOSDEM-2012, I also doubt this is the case for their OpenVPN contributions.

In addition, all the patches sent to the community have been reviewed by a few members of the development team, including James Yonan, before they were merged into the source tree. These patches are basically 3 groups of patches:

1) Documentation patches
These patches adds Doxygen documentation blocks in the source code, on the important pieces of OpenVPN in regards to crypto and some general functionality. These changes should not modify the compiled code in any way. IIRC, this was added already to the OpenVPN 2.2 release. More info about the review of these patches can be found here: http://thread.gmane.org/gmane.network.openvpn.devel/4740/focus=4747

2) Modularisation of the SSL/crypto code
These patches introduced an abstraction layer with a clear API to make it possible to add new SSL/crypto library support in almost a "drop-in" manner. What this means is that the current OpenSSL implementation got moved into separate files and function names were basically changed to use the abstraction layer API. Throughout the OpenVPN all SSL and crypto calls where modified to use this new abstraction layer instead of the native OpenSSL calls. The review process did not reveal any changes of the OpenSSL calls at all, but the review process did confirm that the code which performs the operations was the same when it was moved into new files and into the new abstraction API. This code arrives in OpenVPN v2.3.

3) Adding support for PolarSSL
These patches adds support for PolarSSL, using the new abstraction API. The review process did not go too deep into what PolarSSL does under the hood (that would be to review PolarSSL itself, which is outside the scope of our patch reviews). But it was made sure that the code looked reasonable and is comparable to what the OpenSSL code does. This code also arrives in OpenVPN 2.3.

The review process of all these patches related to documentation, modularisation and PolarSSL is documented here:
https://community.openvpn.net/openvpn/wiki/PolarSSLintegration
You can also here easily find the changes as well here.

In addition, the git tree contains remarks about whom who gave the ACKs for inclusion. We have a fairly strict rule that patches needs to receive a public ACK (either on the mailing list or on the #openvpn-devel IRC channel). Even though, on "obvious" patches which has been lingering around for some weeks without a review and seems reasonable, they might be included without an ACK. But this happens very seldom.

So, in regards to guarantee that no "spy door" or "back door" feature got introduced via these PolarSSL changes, it is difficult to fully guarantee that. It all depends on how clever the developers behind the patches are. Having that said, these patches were not blindly added to the source tree. All of them (100+ patches) were reviewed by more people who can write and understand C code. In addition the most critical patches were also reviewed and ACKed by James Yonan. And if someone finds anything nasty, we will of course take action instantly as soon as we're made aware of it. The best way to get our attention is either to use the openvpn-devel@lists.sourceforge.net mailing list or grab us at #openvpn-devel @ FreeNode.

And thank you very much for being critical and raising such important questions! Even though, it takes time to give arguments to such discussions, it is an important aspect to cover. OpenVPN is a security product, and to use it - you need to be sure you can trust it. But OpenVPN is also open source software, so anyone who are interested are encouraged to participate in reviewing our changes - also after it has been committed or released.

kind regards,

David Sommerseth

palatinux

Dear Dazo,

From our perspective as well-trained security experts, we would like to see that the OpenSSL implementation will stay in OpenVPN for the sake of security. Or that PolarSSL and OpenSSL are replaced by GNUTLS one day (the best crypto engine).

We'll tell you why we don't like PolarSSL and dropped or converted every Linux program that relies on it in Fortress Linux. But we won't discuss the issues we've found with OpenSSL here.

The encryption that PolarSSL provides can be subjected to side-channel attacks because of, for example, the wrongfully implemented or broken PRNG (which can crash OpenVPN), the block cipher modes of operation and predictable/weak Diffie-Hellman exchanges. Next to that, we also found other ways to crash applications that rely on PolarSSL and execute code in memory to get (root) access to a system (buffer overflow).

We tried to discuss this with the PolarSSL developer but we didn't have any cooperation from his side other than waving with his 'FOSS flag', almost demanding us to buy an expensive closed-source license of PolarSSL, lawyers etc. (Though the FOSS doesn't apply to finding / publishing exploits).

Since we have wasted / invested a lot of time in auditing PolarSSL and fixing everything that relies on it, we are now going to put these PolarSSL exploits up for sale. We assume this will frustrate some people, but finding exploits is a very expensive and time-consuming task that will (almost) never be done by spare-time developers (that are often no trained security experts).
I'm sorry that we have to do it this way, but we don't see any other options.

PS. Maybe users should know that the CEO of FOX-IT doesn't object against a government hacking into computer systems, he almost sees it as a must-have:

( Pages are translated from Dutch to English)
http://alturl.com/r4yns
http://alturl.com/ruird

We wish you all good fortune with the new OpenVPN 2.3

pjbakker

I was amazed by Palatinux's post. His reasons for posting are his own. I'm not here to judge. I do want to provide the other side.

TL;DR We cannot claim PolarSSL is without possible vulnerabilities. We are not aware of any and have not been informed of any issues.

In short (longer version below):

We cannot claim PolarSSL is without possible vulnerabilities. I don't think any software can claim that. We are not aware of any known issues and have not been contacted about any possible issues either. We handle every issue brought to us as swiftly as possible. We want to make the best and most secure SSL library possible and will do anything we can to achieve that!

To everybody, including palatinux: We are always open for discussing anything, have never kept anything secret, never closed down any discussion and will never try to hide possible vulnerabilities. Mail us if you want to disclose anything and inform us about it.

I find it perplexing that the same guys that question the integrity of the contributers from Fox-IT who added PolarSSL support to OpenVPN and are also the ones that go on to threaten everybody that they will sell exploits for PolarSSL to the highest bidder.

Paul Bakker
PolarSSL

For people wanting more background:

I was just re-reading all mail conversations I had with Palatinux. They tell a different story than the post above and I think it all stems from one mail I sent about not complying with the GPL and other FOSS licenses.

I can only say from our side, that we have never received any mail from them on the mentioned issues. Even the opposite, when I asked them personally for any pointers / patches they had or knew about on April 3rd 2012, I was told they did not provide patches because of bad experiences with other vendors.

Literal quote in Dutch:

We houden geen rekening meer met andere software ontwikkelaars of bedrijven en we publiceren oplossingen voor lekken of zwakheden (in principe) alleen nog maar tegen betaling of tegendienst. Maar we leveren het wel gratis aan bepaalde groeperingen, wat uiteindelijk een hoop lolz oplevert 🙂
De enige die wij beschermen, zijn onze klanten.

Literal translation to English:

We do not take other software developers or companies into account anymore and (in principle) we only publish solutions for leaks or vulnerabilities in exchange for payment or for favours in return. We do supply these for free to certain groups, which in the end provides us with a lot of lolz 🙂
The only ones we protect, are our customers.

Up till October 4th 2012 Palatinux is a happy user. We added some requested features or changes to the trunk. The praise he gave made us happy and shows the contact up till then.

Literal quote in Dutch:

We hebben trouwens onze eigen OpenVPN + PolarSSL en onze Hiawatha webserver inplemetatie laten testen in Duitsland. Het overtreft nu al OpenVPN-NL en de standaard Hiawatha. Was (haast) niet mogelijk geweest zonder Polar, dus bedankt in ieder geval.

Literal translation to English (translation does not include typo from original):

We have had our own OpenVPN + PolarSSL and our Hiawatha webserver implementation tested in Germany. It already surpasses OpenVPN-NL and the standard Hiawatha. Would (almost) not have been possible without Polar, so thanks at least.

That's quite a jump from their current post, isn't it?

October 4th 2012 is also the turning point as, in the reply to that precise email, I mention to them that their current site and License was and is misrepresenting FOSS IP rights (including Linux Kernel, OpenVPN and at that point PolarSSL). They don't provide source code on the website for their binary only ISO distributions and their Licenses claim to own all copyright on the FOSS they use (including Linux kernel, OpenVpn and at that time PolarSSL) and that this misrepresents the situation. No buying of licenses was suggested, certainly not demanded. Just a mail asking them to change their license text to represent FOSS rights and licenses.

Within one month that single mail culminated in them publicly dropping PolarSSL from FortressLinux without prior contact on why, what vulnerabilities were found and how to fix them and one end result is posting stuff like the one above.

Paul Bakker
PolarSSL

dazo

palatinux wroteDear Dazo,
From our perspective as well-trained security experts, we would like to see that the OpenSSL implementation will stay in OpenVPN for the sake of security. Or that PolarSSL and OpenSSL are replaced by GNUTLS one day (the best crypto engine).

OpenVPN moved over to a more modular approach when PolarSSL support got added. The intention of this was purely so that OpenVPN can be extended further to support other crypto libraries as well. PolarSSL is a good library, especially in the embedded world where OpenSSL consumes a lot of space. But we also didn't want to lock ourselves to only OpenSSL or PolarSSL. So it should be possible to extend OpenVPN to support GNUTLS too.

However, the OpenVPN developers will not put much efforts in implementing GNUTLS support now. But if you or other developers are interested in seeing GNUTLS support, we are open to apply patches which fits into the scheme for crypto modules. But just to have that said as well, there are also NO plans to remove OpenSSL support.

In regards to your experiences and opinions on PolarSSL and its developers, that is not a discussion I want to spend much time on and definitely not here. But I'm happy to see that Paul has responded. I can only say that from my own experience Paul has been very helpful and supportive whenever we've been in contact. If PolarSSL got issues, I trust that the developers and finders are able to figure this out and provide the needed fixes as soon as possible - just as with all other software.

palatinux

Paul,

We have discussed about many PolarSSL issues and shortcommings in the past, which I cannot find anywhere here.

At first hand, PolarSSL looked promissing enough to replace OpenSSL, but when the first weak pnrg issue showed up (https://polarssl.org/tech-updates/security-advisories/polarssl-security-advisory-2011-02), we decided to take that test in Germany > everything seems te be fine again after those patch series.

But after some time we discovered a major, but almost unpatchable issue in PolarSSL. We already gave you enough hints to discover it yourself before you started denying and waving with the FOSS and that close-sourced license of 1300 euro's.

Did you really expect that we would invest so much time and money to fix -your- broken product and give it all away for free to someone who refuses to listen or to improve his product by himself.. and buy a license from you too? And are you going to refund all the server downtime costs PolarSSL gave us until we where finally able to drop it for good?

We warned the OpenVPN community and that is what counts. And Paul, you know what needs to be done and stop protecting yourself over and over again. Everyone makes mistakes and be happy about it that this leak is not yet found by a malicious group or goverment, so you have some time left.

ps. paul, I think you missed our email disclaimer of the emails you have posted here:

"The information contained in this communication is intended to be only
for the addressee. Any use by third parties, and disclosure, copying,
or distribution of this information is prohibited"

palatinux

We've posted our answer here some days ago. But we are still waiting for approval from the OpenVPN forum moderator.

erogravity

I'd like to give an outsider's (of VPN) perspective here about security and government, although the way the thread ended it seems an "unsaid" consensus was reached. I'll go ahead and say it:

At this point in the game with everything we know about the NSA's admissions of how much data they collect, it is 100% that SELinux and any open source project or company supporting, employing, or working with alphabet agencies (almost at any point) has been bought off in some way or another and should not in any way be considered secure computing.

Whether code has hooks or fully operational backdoors is beside the point. Exploits are regularly sold to governments and the more hooks you have, the easier it is to assemble an exploit.

I came here after days of searching for secure Linux kernel implementations free of as many hooks as possible and to see the full extent of open source corruption even extending to VPN's is ...disturbing. That the NSA had hooks into the Linux Kernel since 2.6 (And openly in MS Vista as well) should let you know how far along the game we are.

palatinux, thank you so much for the enlightenment and courage to post this.

smokingwheels

Just my 2 cents worth..
Install Peerblock on your server if you are really scared add all lists..http://www.peerblock.com/
Some configuration is needed to allow private ip's though and OpenVPN.

I hope you are not using a usb stick for internet with win 2000 and ICS see https://forums.openvpn.net/topic12261.html
It would cost me money which I dont have ATM to reproduce and test. Have normal connection now.

I have bound a shoutcast server to clients IP...

PS I have time but no $$$$ my school English is basic though
PPS change your hosts file and add 127.0.0.1 deploy.akamaitechnologies.com plus various other servers they use..there is not much info on them but they store files and content with out your permission on your pcs hdd. I use wireshark to find there server names and then block them..had 2 GB recharge used in 4 days normally last 20- 30 days.

palatinux

We've developed our own alternative to OpenVPN; a more secure and faster VPN solution. So this means that we are not tied to OpenVPN or its security anymore.

I would like to ask if one of your moderators could remove our account from this forum.

Revele

Hmm, thanks for warning, it's really a bit creepy for me and I feel unsafe. I hope FOX-IT turned to be good guys and old good OpenVPN will wirk as perfectly as before.

cleantalkc

The test message, ignore and delete it.

rutukate

Well explained.

Charles99

Thanks for the information. Is it good news or bad news that FOX-IT has jumped in?

Pippin

Depends on tightness of the tin foil hat.

Igor Neo

Everything is much clearer now. Thanks for the explanations