An Interview with OSTIF, the Team Behind the OpenVPN Audit

You’ve just completed your security audit of OpenVPN. Two people worked for almost two months on this project. How does such an audit work?

It actually turned out to be three researchers working a combined 50 days (roughly 1,000 hours) on the security review.

When we plan to audit a piece of software, there is significant work that goes into planning the timing of the audit, who will do the work, and which areas of the software we will cover.

For OpenVPN, we waited until the release of OpenVPN 2.4, which contains some major code changes. We could then evaluate the new features as well as many under-the-hood changes.

Updates that contain significant code changes are good moments to evaluate software, as coding errors can make it through testing, or regressions of end-user features can slip through the cracks.

OpenVPN is unique software in that it is monolithic code with many features that have to stay compatible with older versions. Maintaining legacy compatibility slows down the security review process. We have to navigate a complex web of features, rather than a modular design where the application can be evaluated piece by piece. OpenVPN also relies on two different libraries (OpenSSL and PolarSSL) for cryptography, which means there are two entirely different cryptographic stacks powering the protection.

On top of that, there is OpenVPN 3.0, which is a unique version that is not fully open-source. OpenVPN 3.0 was created because of Apple App Store licensing issues that prevent free software in the store. The OpenVPN 3.0 code is used for OpenVPN Connect for Android and iOS. If we had to evaluate this entire ecosystem, many researchers would need many months to wrestle with all these variations of OpenVPN, and then they would still have to look at all the different network and hardware configurations these different apps may encounter. The complexity and cost would be enormous.

We consulted with experts and worked with the OpenVPN team and QuarksLab to figure out what to focus on. It was decided that OpenVPN 2.4 for Windows and Linux covers the most users and would do the most good. Most commercial VPN providers use OpenVPN 2.4 code for their custom VPN clients because of the licensing structure around it.

We also decided to focus on any cryptography created by OpenVPN itself, and on the security of the application. That means looking for logic errors, memory allocation errors, improper buffer handling, or other improper error-state vulnerabilities.

A separate audit of OpenSSL would let us closely evaluate the cryptography of OpenVPN itself, to ensure that both the cryptography and the application are sound. It is important to create a secure and hard-to-exploit application for users to enjoy.

As for the actual audit process, QuarksLab does an excellent job of documenting the processes and tools used in evaluating software. Our work focuses on planning the scope of the audit and setting achievable goals. We then rally the open-source, security, and privacy communities around the cause to raise the money to do it.

Are there any surprising / remarkable findings from your audit that you could share with us now?

We are in a blackout stage of the audit process for OpenVPN, so I won’t be able to discuss any specifics that people might find in the results, but they will be publicly available very soon. We are waiting on OpenVPN 2.4.2.

What is the rationale behind such an audit? Are you guarding against potential security holes, or do you simply want to take a closer look at software you regularly rely on?

Our strategy as an organization is to cover different areas of security and privacy and to pick widely used applications.

VeraCrypt was a much-needed successor to TrueCrypt, which the community relied on heavily, but the people running the project were relatively unknown and had taken on a large project with complex code. It made logical sense to approach it as our first audit, because we could evaluate the changes in the code that went into TrueCrypt 7.1a and compare them with the current version of VeraCrypt. This narrow scope let us drastically reduce costs and show people that the organization is effective at getting results.

OpenVPN is our first “broad” application audit. It required a much larger budget, but it also had a large community of VPN providers (who are themselves privacy activists). VPN providers care about both their users’ privacy and are directly concerned with the security of OpenVPN, which allowed us to raise funds from both commercial OpenVPN interests and private users at the same time.

OpenSSL is bigger still, but it has industry support all around it, as OpenSSL code (and other libraries derived from it) powers around 70% of the top 1,000,000 websites. This gives us many business interests we can ask for funding to help evaluate OpenSSL 1.1.1, which will be the first version of OpenSSL with the new TLS 1.3 code.

As we move down the list of applications we plan to audit, it becomes harder to raise funds — either because the community surrounding them is smaller, or because there is no major business interest in the application’s success.

We hope that after repeated successes we will be able to secure larger corporate sponsors, which will let us direct funds to these projects more efficiently without relying entirely on small public donations. This will also help us a great deal in building our other programs, which include working with projects to make their applications easier to use, improving testing methods and tools, and creating easy-to-follow guides for the privacy and security software we support.

In short, all of this is currently part of a larger strategy of supporting one application from each major area of privacy and security, and then expanding from there. Our criteria are the perceived strength of the software combined with wide adoption.

For your OpenVPN project, you received support largely from the VPN industry. Did you expect support beyond that? How satisfied are you with this support?

We also received great support from the community, both through word of mouth and through direct donations.

Our goal was exceeded surprisingly quickly, as we initially believed the one-month window we allotted for fundraising would be insufficient. But we reached our goal and raised significantly more than planned within 20 days. That money was set aside for the bug bounty program, which is planned to kick off in the summer / fall.

I was surprised by the community’s positive response and the outpouring of support for the project. It truly was remarkable! I am very pleased with the community’s support for the project, but I was also surprised by the number of larger organizations that did not respond to our inquiries or had no point of contact for their management.

Still, overall the good far outweighs the bad, and we look forward to working with all our supporters on the OpenVPN initiatives and beyond!

You moved from a pooled fundraising model to a direct fundraising model, where you raise funds for each project separately. It seems to have worked well for the OpenVPN project, where the VPN industry happily donates. Do you expect future projects to be funded in a similar way, and how will that work for software projects that don’t have a commercial industry around them, like OTR?

The change in funding model came from community feedback about sticker shock. During our first round of fundraising, we planned a year of activities and then tried to raise the effort through Kickstarter. That led to financial hurdles, such as offering rewards for donations, Kickstarter fees, and payment services skimming money off donations. On top of that, the 8 planned projects combined into a goal in the millions of dollars. As a newcomer to the industry with no track record, the huge sum of money — and after several high-profile Kickstarter failures — it was doomed from the start.

Our change in strategy reduced costs and numbers to the bare minimum and set more achievable goals, but it also requires significantly more work for each fundraiser. We hope that once we build a reputation for accountability and effectiveness, we will be able to secure larger donors, which will let us focus more on executing things and less on directly soliciting donation requests. Larger donations will also have the added advantage of letting us fund less commercial projects like OTR, Nginx, Tunnelblick, and others.

How do you see privacy and security-enhancing technology evolving? Especially with regard to mobile phones and proprietary systems?

We have repeatedly seen through various leaks from government agencies that if the cryptography around information is good, they will not be able to break it at scale.

This fact at the very least disables the “listen to everyone” form of mass surveillance that has become increasingly prevalent over the last several years. As these privacy tools continue to improve and cryptography becomes harder to break and easier to use, we will see significantly increased efforts to attack and compromise devices.

There is evidence of this through the large-scale theft of SIM card keys from Gemalto, huge lists of stolen RSA keys in NSA leaks, backdoors inserted into Cisco and Juniper systems, and so on.

The security community has long called for a “full stack” of open source surrounding the devices that hold our most personal information. The biggest obstacle right now is funding and organizing the support to actually make it happen.

It seems some companies are doing great work on the proprietary side, but we have learned many times that we cannot trust a black box of code. See this month’s heap overflow in iOS: https://googleprojectzero.blogspot.com/2023/04/exception-oriented-exploitation-on-ios.html

Android has many ecosystem-related problems tied to lagging updates, which creates millions of vulnerable devices. Or companies carelessly stop updates for their phones after sales end. Then there are even deeper problems like the vulnerable Broadcom radio firmware that will never be patched, as Project Zero recently demonstrated.

A truly open-source phone is a big ask, but we can certainly try to push the open-source community in the right direction by developing pieces of the puzzle independently. I really hope we can get there, because the current situation is a mess. I’m shocked there isn’t already a Mirai-style botnet based on smartphones knocking over cell towers around the world with data floods.

Apple makes a lot of positive news with its proprietary systems regarding security and privacy. What role do you think open-source projects will play in bringing usable technology to the masses while respecting users’ rights?

Apple has invested enormous resources in building a phone ecosystem that focuses on security. The problem is that Apple does not open up this technology, so we are dealing with the same problem that affects proprietary software like Windows.

We have a black box of millions of lines of code of unknown quality, all interacting with each other in known ways. Apple relies on the inability of malware authors and security researchers to reverse-engineer its code and find flaws. Part of this motivation is to lock the software to the phones, so that iOS can only be installed on genuine Apple hardware. Another motivation is locking the phones to the software, so you can’t buy an iPhone and put an alternative operating system on it — preserving their ability to draw money through the app store with a captive audience.

To be clear, at the moment they are objectively doing a better job than Google when it comes to overall security. The problem is that this black box cannot be trusted. It has bugs, like any software — thousands of bugs. Since this software is proprietary and the source is unavailable, those bugs are waiting to be discovered by Apple’s security team or by someone else in the world who finds them first.

Open-source software can be reviewed. It removes the “just trust me” question that no privacy-minded person can objectively accept.

I hope Google moves in Apple’s direction, as updates would then be enforced on all devices regardless of the vendor, and hardware requirements would have to be tightened for that to happen. I also hope we can open up the source code of Google’s currently closed firmware parts and related drivers, so that we can trust the full stack the phone relies on for security. That would put an open solution in a position to lead the market with good security and privacy practices.

You look at a lot of other people’s code. What common mistakes do you observe? Which bugs are the most common?

I actually don’t do the security reviews myself; those are left to the contracted auditors. But the most common problems are memory management issues and the proper deletion of security-related data when it is no longer in use.

The other big mistake is trying to write your own cryptography. It is wildly complex, and there are many, many ways to defeat cryptography that have been invented over the last several decades. You have to study them carefully and follow many standards to create strong cryptography. Using already vetted libraries avoids this minefield entirely.

Do you have any advice to share with the many coders reading this?

Support a security or open-source initiative. Volunteering your time and knowledge as a coder is extremely valuable, even if you make just one commit a month to a worthy project.

The sum of the community’s skills and time adds up to applications that can change the internet and the world for the better. If you don’t have the skills, make a small recurring donation to an organization that helps build and improve these tools and libraries. I’m not just talking about OSTIF, but the Free Software Foundation or any of those donation buttons you see when you download a piece of open-source software.

You’d be shocked how a few dollars help small projects function and improve. Small contributions add up to a better digital world for all of us.