How ExpressVPN Keeps Its Web Servers Patched and Secure

An ExpressVPN server rising from the ashes.

This article explains ExpressVPN’s approach to security patch management for the infrastructure that runs the ExpressVPN website (not the VPN servers). In general, our approach to security is:

  1. Make systems very hard to hack.
  2. Minimize the potential damage if a system is hypothetically hacked, and acknowledge the fact that some systems cannot be absolutely secure. Usually this starts at the architecture design stage, where we minimize an application’s access.
  3. Minimize the amount of time that a system can remain compromised.
  4. Validate these points with regular pentests, both internal and external.

Security is ingrained in our culture and is the primary concern in all of our work. There are many other topics, such as our secure software development methods, application security, employee processes and training, etc., but they are beyond the scope of this post.

Here we explain how we achieve the following:

  1. Make sure all servers are fully patched within no more than 24 hours of CVE publications.
  2. Make sure no server is in use for more than 24 hours, thereby imposing an upper limit on the amount of time an attacker can have persistence.

We achieve both goals through an automated system that rebuilds servers starting from the OS with all the latest patches, and destroys them at least once every 24 hours.

Our goal with this article is to be useful to other engineers facing similar problems, and to provide transparency into ExpressVPN’s operations for our customers and the media.

How we use Ansible playbooks and Cloudformation

ExpressVPN’s web infrastructure is hosted on AWS (unlike our VPN servers, which run on dedicated hardware), and we make heavy use of its features to make rebuilds possible.

All of our web infrastructure is provisioned with Cloudformation, and we try to automate as many processes as possible. However, we find working with raw Cloudformation templates rather unpleasant due to the need for repetition, general poor readability, and the limitations of JSON or YAML syntax.

To mitigate this, we use a DSL called cloudformation-ruby-dsl, which lets us write template definitions in Ruby and export Cloudformation templates to JSON.

In particular, the DSL allows us to write user-data scripts as regular scripts, which are automatically converted to JSON (rather than going through the painful process of turning every line of a script into a valid JSON string).

A generic Ansible role called cloudformation-structure handles rendering the actual template to a temporary file, which is then used by the Ansible cloudformation module:

- name: 'render {{ component }} cloudformation stack json'
  shell: ruby "{{ template_name | default(component) }}.rb" expand --stack-name {{ stack }} --region {{ aws_region }} > {{ tempfile_path }}
  args:
    chdir: ../cloudformation/templates
  changed_when: false

- name: 'create / update {{ component }} stack'
  cloudformation:
    stack_name: '{{ stack }}-{{ xv_env_name }}-{{ component }}'
    state: present
    region: '{{ aws_region }}'
    template: '{{ tempfile_path }}'
    template_parameters: '{{ template_parameters | default({}) }}'
    stack_policy: '{{ stack_policy }}'
  register: cf_result

In this playbook, we call the cloudformation-structure role several times with different component variables to create multiple Cloudformation stacks. For example, we have a network stack that defines the VPC and related resources, and an application stack that defines the auto-scaling group, launch configuration, lifecycle hooks, and so on.

We then use a somewhat ugly but useful trick to turn the output of the cloudformation module into Ansible variables for subsequent roles. We have to use this approach because Ansible does not allow creating variables with dynamic names:

- include: _tempfile.yml
- copy:
    content: '{{ component | regex_replace("-", "_") }}_stack: {{ cf_result.stack_outputs | to_json }}'
    dest: '{{ tempfile_path }}.json'
  no_log: true
  changed_when: false

- include_vars: '{{ tempfile_path }}.json'

Updating the EC2 auto-scaling group

The ExpressVPN website is hosted on several EC2 instances in an auto-scaling group behind an application load balancer, which allows us to destroy servers without downtime, as the load balancer can drain existing connections before an instance is terminated.

Cloudformation orchestrates the entire rebuild, and we run the Ansible playbook described above every 24 hours to rebuild all instances, using the UpdatePolicy AutoScalingRollingUpdate attribute of the AWS::AutoScaling::AutoScalingGroup resource.

When simply re-running without any changes, the UpdatePolicy attribute is not used — it is only invoked under special circumstances, as described in the documentation. One such circumstance is updating the auto-scaling launch configuration — the template the auto-scaling group uses to launch EC2 instances — which includes the EC2 user-data script that runs when a new instance is created:

resource 'AppLaunchConfiguration', Type: 'AWS::AutoScaling::LaunchConfiguration',
  Properties: {
    KeyName: param('AppServerKey'),
    ImageId: param('AppServerAMI'),
    InstanceType: param('AppServerInstanceType'),
    SecurityGroups: [
      ref('SecurityGroupApp'),
    ],
    IamInstanceProfile: param('RebuildIamInstanceProfile'),
    InstanceMonitoring: true,
    BlockDeviceMappings: [
      {
        DeviceName: '/dev/sda1', # root volume
        Ebs: {
          VolumeSize: param('AppServerStorageSize'),
          VolumeType: param('AppServerStorageType'),
          DeleteOnTermination: true,
        },
      },
    ],
    UserData: base64(interpolate(file('scripts/app_user_data.sh'))),
  }

If we make any update to the user-data script, even a comment, the launch configuration will be considered changed, and Cloudformation will update all instances in the auto-scaling group to match the new launch configuration.

Thanks to cloudformation-ruby-dsl and its interpolation feature, we can use Cloudformation references in the app_user_data.sh script:

readonly rebuild_timestamp="{{ param('RebuildTimestamp') }}"

This procedure ensures our launch configuration is fresh on every rebuild run.

Lifecycle hooks

We use the auto-scaling lifecycle hook features to make sure our instances are fully provisioned and pass the required health checks before they are put into service.

Using lifecycle hooks lets us have the same instance lifecycle both when an update is triggered by Cloudformation and when an auto-scaling event occurs (for example, when an instance fails the EC2 health check and is terminated). We do not use cfn-signal and the auto-scaling WaitOnResourceSignals policy, since those only apply when Cloudformation triggers an update.

When the auto-scaling group creates a new instance, the EC2_INSTANCE_LAUNCHING lifecycle hook fires, and it automatically moves the instance into the Pending:Wait state.

Once the instance is fully configured, it curls its own health-check endpoints from the user-data script. As soon as the health checks show the application is healthy, we trigger the CONTINUE action for that lifecycle hook, so the instance attaches to the load balancer and starts serving traffic.

If the health checks fail, we trigger the ABANDON action, which terminates the unhealthy instance, and the auto-scaling group launches another one.

Besides failing the health checks, our user-data script can fail at other points — for example, if temporary connectivity issues prevent software installation.

We want the creation of a new instance to fail as soon as we know it will never become healthy. To do this, we set an ERR trap in the user-data script together with set -o errtrace to invoke a function that sends the lifecycle ABANDON action, so a failing instance can terminate as quickly as possible.

User-data scripts

The user-data script is responsible for installing all the required software on the instance. We have successfully used Ansible for provisioning instances and Capistrano for application deployment for a long time, so we use them here as well, which allows for minimal differences between regular deployments and rebuilds.

The user-data script checks out our application repository from Github, which includes the Ansible bootstrap scripts, then runs Ansible and Capistrano pointed at localhost.

When checking out the code, we must be sure that the currently deployed version of the application is the one deployed during the rebuild. The Capistrano deployment script includes a task that updates a file on S3 which stores the currently deployed commit SHA. When a rebuild happens, the system picks the commit to be deployed from this file.

Software updates are applied by automatically running an unattended upgrade in the foreground with the unattended-upgrade -d command. Once it completes, the instance reboots and runs the health checks.

Working with secrets

The server requires temporary access to secrets (such as the Ansible vault password), which are retrieved from the EC2 parameter store. The server can access the secrets only for a short time during the rebuild. After fetching them, we immediately replace the original instance profile with another one that only has access to the resources needed to run the application.

We want to avoid storing any secrets in the instance’s persistent storage. The only secret we keep on disk is the Github SSH key, but not its passphrase. We also do not store the Ansible vault password.

However, we need to pass these passphrases to SSH and Ansible respectively, and that is only possible in interactive mode (i.e., the utility prompts the user to enter passphrases manually) for a good reason — if the passphrase is part of a command, it is saved in the shell history and can be visible to all users on the system if they run ps. We use the expect utility to automate interaction with these tools:

expect <<EOF
cd ${repo_dir}
spawn make ansible_local env=${deploy_env} stack=${stack} hostname=${server_hostname}
set timeout 2
expect "Vault password"
send "${vault_password}\r"
set timeout 900
expect {
  "unreachable=0 failed=0" {
    exit 0
  }
  eof {
    exit 1
  }
  timeout {
    exit 1
  }
}
EOF

Triggering the rebuild

Since we trigger the rebuild by running the same Cloudformation script that is used to create / update our infrastructure, we must make sure we don’t accidentally update some part of the infrastructure that should not be updated during a rebuild.

We achieve this by setting a restrictive stack policy on our Cloudformation stacks so that only the resources needed for the rebuild get updated:

{
  "Statement" : [
    {
      "Effect" : "Allow",
      "Action" : "Update:Modify",
      "Principal": "*",
      "Resource" : [
        "LogicalResourceId/*AutoScalingGroup"
      ]
    },
    {
      "Effect" : "Allow",
      "Action" : "Update:Replace",
      "Principal": "*",
      "Resource" : [
        "LogicalResourceId/*LaunchConfiguration"
      ]
    }
  ]
}

When we need to make real infrastructure updates, we have to manually update the stack policy to allow explicitly updating those resources.

Because our server names and IP addresses change every day, we have a script that updates our local Ansible inventories and SSH configurations. It discovers instances via the AWS API by tags, renders inventory and config files from ERB templates, and adds the new IP addresses to SSH known_hosts.

ExpressVPN follows the highest security standards

Rebuilding servers protects us against a specific threat: attackers gaining access to our servers through a kernel or software vulnerability.

However, it is only one of the many ways we ensure the security of our infrastructure, including, among other things, regular security audits and ensuring that critical systems are not accessible from the internet.

In addition, we make sure all our code and internal processes meet the highest security standards.