Introduction

Both Arch Linux and Rust have open-sourced their infrastructure code. So how do they keep sensitive information secure while making their infrastructure code public?

Their implementations can be found here:

Arch Linux uses ansible-vault and GnuPG to manage sensitive information, including Terraform access keys, secret keys, and tokens.

The Rust Team uses AWS STS for permission management and AWS SSM to manage sensitive information in Ansible and Terraform. Terraform state is stored on S3.

Arch Linux Team’s Approach

Ansible

Arch Linux achieves this through the vault_identity_list setting in ansible.cfg.

vault_identity_list is a comma-separated list that accepts either files containing passwords or executable files (which must have execute permission). It looks like this:

default@misc/vault-keyring-client.sh,super@misc/vault-keyring-client.sh

This means the executables are called in the following form:

vault-keyring-client.sh --vault-id default
vault-keyring-client.sh --vault-id super

The executable script simply needs to print the key to standard output. This enables approaches such as:

  • Encrypting the key with GnuPG into a file, storing it on GitLab, and using a bash wrapper around GnuPG for decryption
  • Storing the key in 1Password and using a bash script wrapper around the 1Password CLI for decryption

The Arch Linux team uses the first approach. See:

vault-keyring-client.sh

#!/bin/sh

readonly vault_password_file_encrypted="$(dirname $0)/vault-$2-password.gpg"

# flock used to work around "gpg: decryption failed: No secret key" in tf-stage2
# would otherwise need 'auto-expand-secmem' (https://dev.gnupg.org/T3530#106174)
flock "$vault_password_file_encrypted" \
  gpg --batch --decrypt --quiet "$vault_password_file_encrypted"

Since vault_identity_list accepts multiple executables, there can be multiple keys, which means the repository may contain ciphertexts encrypted with different keys. When viewing with:

ansible-vault view vault_filename.yml

All keys are automatically tried.

When creating encrypted content, you specify which vault-id’s key to use for encryption:

ansible-vault create --encrypt-vault-id <vault-id> vault_filename.yml

GnuPG-encrypted files support decryption by multiple recipients. The Arch Linux project stores different ansible-vault keys in separate files, each encrypted with GnuPG. See the following files in misc, which contain the Ansible Vault keys:

GnuPG controls who can decrypt the GnuPG ciphertext and obtain the ansible-vault keys, thereby implementing access control. In summary:

  • Sensitive information is classified by encrypting it with different Ansible Vault keys.
  • GnuPG controls who can decrypt which GPG ciphertext, thereby managing individual permissions.

Their developers’ GnuPG public keys are typically uploaded to GnuPG key servers. The public key IDs can be found here: root_access

The Arch Linux project also uses Terraform to manage IAM (Identity and Access Management) and infrastructure (such as cloud servers and DNS records). Their IAM solution is Keycloak, and they use Hetzner Cloud for infrastructure.

IaC (Infrastructure as Code) generally requires access keys and secret keys. For this, they use a Python script: get_key.py

This script retrieves a specific key’s value from any Ansible Vault file and outputs it to standard output in JSON or other formats.

Terraform

The backend uses PostgreSQL. When first setting up the repository, you run the following:

terraform init -backend-config="conn_str=postgres://terraform:$(../misc/get_key.py ../group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org?sslmode=verify-full"

This saves the PostgreSQL credentials to a local terraform.tfstate file (which is included in .gitignore). Since it still uses get_key.py to retrieve sensitive information, it effectively relies on the same ansible-vault approach.

Once the user has backend access, the Terraform code uses the external provider.

This provider can invoke external executables and store the results in the backend. For example:

data "external" "vault_hetzner" {
  program = [
    "${path.module}/../misc/get_key.py", "${path.module}/../misc/vaults/vault_hetzner.yml",
    "hetzner_cloud_api_key",
    "hetzner_dns_api_key",
    "--format", "json"
  ]
}

# Some unrelated content omitted

provider "hcloud" {
  token = data.external.vault_hetzner.result.hetzner_cloud_api_key
}

provider "hetznerdns" {
  apitoken = data.external.vault_hetzner.result.hetzner_dns_api_key
}

As you can see, it still calls the get_key.py script – the same ansible-vault approach throughout.

This is how the Arch Linux team manages sensitive information in their infrastructure.

Rust Team’s Approach

Terraform

The Rust Team primarily uses the AWS CLI. By running:

eval $(~/PATH/TO/SIMPLEINFRA/aws-creds.py)

It accesses the STS service. After 2FA authentication, the local shell gains temporary credentials.

Since the backend uses S3, once temporary credentials are obtained and the appropriate policies are in place, the infrastructure can be managed.

Ansible

The Ansible setup uses the Ansible Lookup Module: aws_ssm.

aws_ssm is a module that interacts with the AWS Systems Manager service, which provides the SSM Parameter Store. Ansible’s sensitive information is stored as SSM parameters on AWS.

ssm_all: "{{ lookup('aws_ssm', '/prod/ansible/all/', region='us-west-1', shortnames=true, bypath=true, recursive=true) }}"

vars_papertrail_url: "{{ ssm_all['papertrail-url'] }}"

Variables are then accessed as needed using Jinja syntax in a key-value fashion.

This is how the Rust Team manages sensitive information in their infrastructure.