Deploying Vault on VMs with Let's encrypt! TLS certs

In this get started tutorial, we install the vault container in a VM and configure a TLS certificates with let's encrypt.

Prerequisites

Throughout this tutorial, we assume docker and docker-compose are installed, and the user has permission to download the vault container from the repo. We further assume access to a DNS server in order to pass the ACME let's encrypt challenge.

Getting Started

Setting up DNS record

Please consult your DNS administrator. Add an A or AAA to point the VM's IP address to the domain mydomain.com.

Looking ahead the DNS record will serve the ACME challenge.

Creating shared folders

You may want to create folder for shared mounts between the vault and certbot co

# ./data stores encrypted vault data
mkdir data
# ./config stores the vault config file
mkdir config
# ./cert stores the TLS key files 
mkdir cert

Configuring the Vault

Create a file config.json and store the file in folder ./config. Below some example configurations.

Filesystem

storage "file" {
  path = "/etc/vault/data"
}

listener "tcp" {
    address = "0.0.0.0:8200"
    tls_disable = 0
    tls_cert_file = "/etc/letsencrypt/live/vault/fullchain.pem"
    tls_key_file = "/etc/letsencrypt/live/vault/privkey.pem"
}

disable_mlock = true
ui = true

api_addr = "https://0.0.0.0:8201"

Raft Cluster

storage "raft" {
  path    = "./vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
  tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
}

disable_mlock = true
ui = true

api_addr = "https://0.0.0.0:8200"
cluster_addr = "https://0.0.0.0:8201"

Within the configuration file, there are two primary configurations:

• storage - This is the physical backend that Vault uses for storage. Up to this point the dev server has used in memory (inmem), but the example above uses the filesystem (file) and integrated Storage (raft), a much more production-ready backend.

• listener - One or more listeners determine how Vault listens for API requests. The example above listens on 0.0.0.0 port 8200. In your environment set VAULT_ADDR=https://mydomain.com so the Vault client will connect without TLS.

Composing the Vault and Certbot

Run the docker-compose.yml and replace flags --email and --domain with the email address and domain name of your choice.

File Storage

services:
  certbot:
    image: certbot/certbot:latest
    container_name: certbot
    ports:
      - "80:80"
    command: >-
             certonly --cert-name vault --standalone --reinstall
             --agree-tos --no-eff-email 
             --email [e-mail] -d mydomain.com 
             --dry-run
    volumes:
      - ./certs:/etc/letsencrypt/
  vault:
    image: harbor.enclaive.cloud/enclaive-dev/vault:latest
    container_name: vault
    ports:
      - "443:8200"
    depends_on:
      certbot:
        condition: service_completed_successfully
    restart: always
    volumes:
      - ./data/:/etc/vault/data/
      - ./config/:/etc/vault/config/
      - ./certs:/etc/letsencrypt/
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -config /etc/vault/config/config.json

Raft Storage

services:
  certbot:
    image: certbot/certbot:latest
    container_name: certbot
    ports:
      - "80:80"
    command: >-
             certonly --cert-name vault --standalone --reinstall 
             --agree-tos --no-eff-email 
             --email [e-mail] -d mydomain.com 
    volumes:
      - ./certs:/etc/letsencrypt/
  vault:
    image: harbor.enclaive.cloud/enclaive-dev/vault:latest
    container_name: vault
    ports:
      - "443:8200"
      - "8201:8201"
    depends_on:
      certbot:
        condition: service_completed_successfully
    restart: always
    volumes:
      - ./data:/etc/vault/data/
      - ./config:/etc/vault/config/
      - ./certs:/etc/letsencrypt/
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -config /etc/vault/config/config.json

Within the docker-compose file, there are some configurations:

• --cert-name: Pass this name to specify a particular certificate.

• --standalone: Use standalone mode to obtain a certificate for any server. This plugin needs to bind to port 80 in order to perform domain validation.

• --reinstall: If the requested certificate matches an existing certificate, always keep the existing one until it is due for renewal.

In non-production use we recommend to enable the following flags to avoid rate limits:

• --dry-run : Simulates the certificate generation. No keys are generated. This flag is useful in testing the configuration and the DNS challenge.

• --staging : Uses the Let's Encrypt staging server to obtain or revoke test (invalid) certificates.

Initializing the Vault

Initialization is the process of configuring Vault. This only happens once when the server is started against a new backend that has never been used with Vault before. When running in HA mode, this happens once per cluster, not per server. During initialization, the encryption keys are generated, unseal keys are created, and the initial root token is created.

Launch a new terminal session, and set VAULT_ADDR environment variable.

export VAULT_ADDR='https://mydomain.com'

To initialize Vault use vault operator init. This is an unauthenticated request, but it only works on brand new Vaults without existing data:

vault operator init

Example output:

Unseal Key 1: 4jYbl2CBIv6SpkKj6Hos9iD32k5RfGkLzlosrrq/JgOm
Unseal Key 2: B05G1DRtfYckFV5BbdBvXq0wkK5HFqB9g2jcDmNfTQiS
Unseal Key 3: Arig0N9rN9ezkTRo7qTB7gsIZDaonOcc53EHo83F5chA
Unseal Key 4: 0cZE0C/gEk3YHaKjIWxhyyfs8REhqkRW/CSXTnmTilv+
Unseal Key 5: fYhZOseRgzxmJCmIqUdxEm9C3jB5Q27AowER9w4FC2Ck

Initial Root Token: s.KkNJYWF5g0pomcCLEmDdOVCW

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key (previously known as master key).
Without at least 3 key to reconstruct the root key, Vault will remain
permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Initialization outputs two incredibly important pieces of information: the unseal keys and the initial root token. This is the only time ever that all of this data is known by Vault, and also the only time that the unseal keys should ever be so close together.

For the purpose of this getting started tutorial, save all of these keys somewhere, and continue. In a real deployment scenario, you would never save these keys together. Instead, you would likely use Vault's PGP and Keybase.io support to encrypt each of these keys with the users' PGP keys. This prevents one single person from having all the unseal keys. Please see the documentation on using PGP, GPG, and Keybase for more information.

Seal/Unseal

Every initialized Vault server starts in the sealed state. From the configuration, Vault can access the physical storage, but it can't read any of it because it doesn't know how to decrypt it. The process of teaching Vault how to decrypt the data is known as unsealing the Vault.

Unsealing has to happen every time Vault starts. It can be done via the API and via the command line. To unseal the Vault, you must have the threshold number of unseal keys. In the output above, notice that the "key threshold" is 3. This means that to unseal the Vault, you need 3 of the 5 keys that were generated.

Vault does not store any of the unseal key shards. Vault uses an algorithm known as Shamir's Secret Sharing to split the root key into shards. Only with the threshold number of keys can it be reconstructed and your data finally accessed.

Begin unsealing the Vault:

vault operator unseal

Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       d3d06528-aafd-c63d-a93c-e63ddb34b2a9
Version            1.7.0
Storage Type       raft
HA Enabled         true

After pasting in a valid key and confirming, you see that Vault is still sealed, but progress is made. Vault knows it has 1 key out of 3. Due to the nature of the algorithm, Vault doesn't know if it has the correct key until the threshold is reached.

Also notice that the unseal process is stateful. You can go to another computer, use vault operator unseal, and as long as it's pointing to the same server, that other computer can continue the unseal process. This is incredibly important to the design of the unseal process: multiple people with multiple keys are required to unseal the Vault. The Vault can be unsealed from multiple computers and the keys should never be together. A single malicious operator does not have enough keys to be malicious.

Continue with vault operator unseal to complete unsealing the Vault. To unseal the vault you must use three different unseal keys, the same key repeated will not work.

vault operator unseal

Unseal Key (will be hidden):
 ...

As you use keys, as long as they are correct, you should soon see output like this:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.7.0
Storage Type            raft
Cluster Name            vault-cluster-0ba62cae
Cluster ID              7d49e5fd-a1a4-c1d1-55e2-7962e43006a1
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24

When the value for Sealed changes to false, the Vault is unsealed.

Feel free to play around with entering invalid keys, keys in different orders, etc. in order to understand the unseal process.

Finally, authenticate as the initial root token (it was included in the output with the unseal keys).

vault login
Token (will be hidden):

Enter the token value when prompted.

Example output:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.spAZOi7SlpdFTNed50sYYCIU
token_accessor       OevFmMXjbmOCQ8rSubY84vVp
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

As a root user, you can reseal the Vault with vault operator seal. A single operator is allowed to do this. This lets a single operator lock down the Vault in an emergency without consulting other operators.

When the Vault is sealed again, it clears all of its state (including the encryption key) from memory. The Vault is secure and locked down from access.