MariaDB root admin password provisioning on Azure DCXas_v5 VM

In this self-contained tutorial we show how to provision a MariaDB container running in a confidential "buckypaper" VM on Azure. The tutorial easily transfers to any other CSP with buckypaper VMs.

In this tutorial, you will learn how to:

Configure the vHSM to attest a cVM.
Set up the vHSM to generate a high-entropy admin password.
Configure the enclaivelet to attest the cVM and securely retrieve the password.
Use cloud-config to automate these processes during VM startup.

Introduction

Setting up a cloud environment and configuring the network can be fraught with challenges. Given the complexity of cloud systems and the pressure to deliver products quickly, issues are almost inevitable.

Challenge

When cloud workloads fail to authenticate properly, they create critical vulnerabilities that can lead to significant security breaches. Without proper authentication, unauthorized users, malicious actors, or compromised applications may gain unrestricted access to sensitive data, services, and resources. This can result in data theft, manipulation, or deletion, leading to operational disruptions and financial losses. Furthermore, the absence of authentication increases the risk of privilege escalation, enabling attackers to move laterally within a network, compromise additional systems, and launch further attacks. This also complicates incident detection and response, making it difficult to trace malicious activities. In summary, missing authentication in cloud workloads compromises trust, compliance, and overall cloud security.

Solution

Traditionally, only users had identities, which led to the development of identity management services in the cloud. These services ensure that users authenticate with an identity provider that centrally manages access permissions and grants access to specific resources. For example, a user might authenticate with an identity management service (such as Entra on Azure, formerly known as Active Directory), which verifies permissions and grants access to a service like Key Vault with the appropriate capabilities (e.g., administrator role). Best practices emphasize the principle of least privilege, ensuring that users only have the access necessary to perform their tasks. Centralized management of user access also minimizes credential sprawl and reduces the risk of credential leakage by avoiding unnecessary storage and sharing.

Confidential computing introduces a new approach where workloads running in enclaves are assigned their own identity. With Nitride, the principles of identity management are extended to workloads. Unlike user authentication, a Buckypaper VM attests (where "attestation" for workloads is akin to "authentication" for users) to the workload identity provider, Nitride, to verify its authorization to access the Vault key management service. In Nitride, a policy defines how workload identities are verified and what access privileges they receive.

Prerequisites

For this tutorial, it is assumed that you have permission to create a confidential "buckypaper" VM on Azure. We have selected a cVM from the DCXas_v5 family, which supports AMD SEV-SNP. Alternative configuration options are available in the accompanying table. Additionally, it is assumed that the vHSM is correctly running on this cVM and is accessible via the /auth/ratls path.

Getting-Started Blueprint

Any use case involving the secret provisioning of attested workloads generally includes one or more of the following steps:

Create the enclave identity

Creation of the enclave identities has to be done once for the cloud configuration

Create the Namespace (Optional)

In certain scenarios, it can be helpful to store secrets within namespaces. To achieve this, you can create a namespace called some-ns by running the following command:

vault namespace create some-ns

Configure a Platform Artifact

In the example below, the enclave identity is defined by a platform artifact, as described in the platform.json file. This artifact details an AMD SEV-SNP enabled platform with VCEK attestation support.

In more complex cases, the enclave identity may consist of a combination of platform, firmware, workload, and meta artifacts.

{
  "type": "platform",
  "name": "amd-sev-snp-milan-1.5.4",
  "values": {
    "root_of_trust": "cert-chain-milan-vcek", # refers to the PEM certificate
    "SEV API firmware": "1.5.4"  
  }
}

Run the command to create the platform identity

vault write auth/ratls/identities @platform.json # only required if policy is set in create.json

Define an attestation verification policy

We need to tell Nitride next how to verify an attestation. To this end we define policy.json. The canonical example below enforces the policy that platforms we previously defined under name amd-sev-snp-milan with firmware version 1.5.3 <= 2.0.0 pass verification.

{
  "name": "Azure_DC2as_v5",
  "description": "Azure AMD SEV-SNP Milan (EPYC3) platform-only verifcation policy",
  "namespace": "some-ns", 
  "provider": "azure-sev-snp-vtpm",
  "events": "test-webhook",
  "policy": {
    "platform": {
      "selector": {
        "name": "amd-sev-snp-milan-1.5.4",
      }
    },
    "firmware": null,
    "workload": null,
    "metadata": null
  }
}

Run the command to create the policy

vault write auth/ratls/attestations @policy.json

Example output:

Key            Value
---            -----
created        1724793304
description    Azure AMD SEV-SNP Milan (EPYC3) platform-only verifcation policy
name           Azure_DC2as_v5
namespace      some-ns
nonce          n/a
policy         map[firmware:<nil> metadata:<nil> platform:map[identity:map[created:1724793285 name:amd-sev-snp-milan type:platform values:map[firmware:^1.0.0 root_of_trust:cert-chain-milan-vcek]] selector:map[name:amd-sev-snp-milan values:map[firmware:^1.0.0]]] workload:<nil>]
provider       azure-sev-snp-vtpm
report         <nil>
totp           n/a
updated        0
uuid           5432087a-726a-4600-b81c-13988c96957a
events         test-webhook

Note down the uuid. It will be instructive to parameterize the enclaivelet environment variable ENCLAIVE_WORKLOAD.

Create TOTP to write the reference attestation report

So far we created the policy with an empty report=<nil> property. We need to fill the property with a value containing the attestation measurements. They serve reference values and are needed to implement the attestation verification.

Create a time-based one-time-password with the permission to write or overwrite the reportproperty.

vault write -force auth/ratls/attestations/5432087a-726a-4600-b81c-13988c96957a/totp

Example output:

Key            Value
---            -----
created        1724798288
description    Azure AMD SEV-SNP Milan (EPYC3) platform-only verifcation policy
name           Azure_DC2as_v5
namespace      some-ns
nonce          n/a
policy         map[firmware:<nil> metadata:<nil> platform:map[identity:map[created:1724798274 name:amd-sev-snp-milan type:platform values:map[firmware:^1.0.0 root_of_trust:cert-chain-milan-vcek]] selector:map[name:amd-sev-snp-milan values:map[firmware:^1.0.0]]] workload:<nil>]
provider       azure_sev-snp_vtpm
report         <nil>
totp           H-CVSoyNTF797S4kjq7GOehYmJ6j46bgTVrGWu1eYDk=
updated        1724799624
uuid           5432087a-726a-4600-b81c-13988c96957a
webhook        test-webhook

Note down the totp. It will be instructive to parameterize the enclaivelet environment variable ENCLAIVE_TOKEN

Activate the buckypaper engine

To register the buckypaper extension vault-plugin-secrets-dkv, use the following command with the correct SHA-256 digest:

vault plugin register -sha256=<digest> secret vault-plugin-secrets-dkv

To verify successful registration, run the command below and look for vault-plugin-secrets-dkv in the list:

vault plugin list | grep vault-plugin-secrets-dkv

We'll activate two endpoint instances of the vault-plugin-secrets-dkv extension

vault secrets enable -namespace=some-ns -path=buckypaper vault-plugin-secrets-dkv

Run the command below to confirm that the endpoint has been enabled correctly:

vault secrets list -namespace=some-ns

Example output:

Path           Type                        Accessor                             Description
----           ----                        --------                             -----------
buckypaper/    vault-plugin-secrets-dkv    vault-plugin-secrets-dkv_86fad1be    n/a
cubbyhole/     ns_cubbyhole                ns_cubbyhole_8ea08522                per-token private secret storage
identity/      ns_identity                 ns_identity_45775dad                 identity store
sys/           ns_system                   ns_system_ae8d65fd                   system endpoints used for control, policy and debugging

Set up the cVM with the enclaivelet

Download the enclaivelet

The enclaivelet actually consists of the two binaries enclaivelet and provision. The first is the attesation shim. The aim is to establish a secure session with Nitride, retrieve an attestation report, and redeem the report to obtain the authentication token. The latter as the name suggests, establishes a secure session with Vault, redeems the previously auth token to get access to secrets, and provisions the secrets into the workload.

Download binaries enclaivelet and provision and make them executable:

wget -c http://127.0.0.1:8200/static/{enclaivelet, provision}
chmod +x enclaivelet provision

Configure the enclaivelet

Set NITRIDE_WORKLOAD to point to the uuid during the policy creation

export NITRIDE_WORKLOAD=5432087a-726a-4600-b81c-13988c96957a

Set NITRIDE_PROVIDER to the attestation method provided by the cVM

export NITRIDE_PROVIDER=azure-sev-snp-vtpm

Set NITRIDE_FEATURES to source a random password in the environment variable MARIADB_ROOT_PASSWORD

Loads (buckypaper/:workload/env/PASSWORD) and stores MARIADB_ROOT_PASSWORD=response.data.value in /tmp/environment.

export NITRIDE_FEATURES=env:PASSWORD:MARIADB_ROOT_PASSWORD

Set the vault and nitride address

export NITRIDE_ADDR=http://127.0.0.1:8200
export VAULT_ADDR=http://127.0.0.1:8200

In case of initialisation or update of the attestation object, set ENCLAIVE_TOTP to contain a valid time-based one-time password. It gives the right to (over)write the attestation report property

export NITRIDE_TOTP=H-CVSoyNTF797S4kjq7GOehYmJ6j46bgTVrGWu1eYDk=

Create the attestation

Run the enclaivelet with the above environment variables

./enclaivelet

After successful attestation Nitride issues the ENCLAIVE_TOKEN to grant access to the buckypaper secrets engine in charge of generating the admin password.

Provision the password from Vault

It remains to contact the buckypaper secrets engine with the request to generate a high-entropy password, retrieve it over a secure channel, and source it into the enclave

./provision

Cloud-Init: Putting it all together

Cloud-init is a popular method for customizing a Linux VM during its initial boot. It allows you to automate tasks like installing packages, configuring users, setting up security, and writing files. Since cloud-init runs as part of the first boot process, no additional agents or steps are needed to apply configurations.

One of cloud-init's strengths is its compatibility across different distributions. Instead of specifying commands like apt-get install or yum install for package installation, you simply define a list of packages, and cloud-init automatically uses the appropriate package manager for the selected distribution.

Create cloud-init config file

At your bash prompt or in the Cloud Shell, create a file named cloud-init.txt and paste the following configuration. For example, you can type sensible-editor cloud-init.txt to create the file and choose from the available editors. Ensure that the entire cloud-init file is copied correctly, paying special attention to the first line:

#cloud-config
runcmd:
  - |
    (
    set -eu

    # Variables
    export MARIA_PASSWORD=${maria_password}
    export VERSION=10.8.2

    # Update packages and install necessary dependencies
    sudo apt-get update
    sudo apt-get install -y ca-certificates curl gnupg

    # Add the official Docker GPG key
    sudo install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    sudo chmod a+r /etc/apt/keyrings/docker.gpg

    # Add the official Docker repository
    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
      $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

    # Update packages and install Docker Engine, Docker CLI, and Containerd
    sudo apt-get update
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

    # Start and enable Docker
    sudo systemctl enable docker
    sudo systemctl start docker

    # MARIA
    sudo docker run --name mariadb -d \
    -e MARIADB_ROOT_PASSWORD=$MARIA_PASSWORD \
    -v /var/lib/mysql:/var/lib/mysql \
    -p 3306:3306 \
    mariadb:$VERSION
    
    # Variables
    export ENCLAIVE_PROVIDER=azure_sev-snp_vtpm 
    export ENCLAIVE_WORKLOAD=5432087a-726a-4600-b81c-13988c96957a
    export ENCLAIVE_TOTP=H-CVSoyNTF797S4kjq7GOehYmJ6j46bgTVrGWu1eYDk=
    export ENCLAIVE_NITRIDE=https://myvhsmdomain.com
    export ENCLAIVE_KEYSTORE=hhttps://myvhsmdomain.com
    export ENCLAIVE_FEATURES=env:PASSWORD:MARIADB_ROOT_PASSWORD

    COMMAND="curl -s -o"

    $COMMAND client "$ENCLAIVE_NITRIDE/static/enclaivelet"
    $COMMAND provision "$ENCLAIVE_NITRIDE/static/provision"

    chmod +x client provision
    ./client
    ) >enclaive.log 2>&1

Create a confidential "buckypaper" VM

Before creating a cVM, start by creating a resource group using az group create. The following example sets up a resource group named myResourceGroupAutomate in the eastus location:

az group create --name myResourceGroupAutomate --location eastus

Next, create a VM using az vm create, and pass in your cloud-init configuration file with the --custom-data parameter. If the cloud-init.txt file is saved outside of your current directory, be sure to provide the full path. The following example creates a VM named myAutomatedVM:

az vm create \
    --resource-group myResourceGroupAutomate \
    --name myAutomatedVM \
    --image Ubuntu2204 \
    --admin-username azureuser \
    --generate-ssh-keys \
    --custom-data cloud-init.txt

It may take a few minutes for the VM to be created, packages to install, and the app to start. Background tasks continue running after the Azure CLI returns to the prompt, so accessing the app might take a couple of additional minutes. Once the VM is created, note the publicIpAddress displayed by the Azure CLI. This IP address allows you to access the Node.js app via a web browser.

To allow web traffic to reach your VM, open port using az vm open-port:

az vm open-port --port <port> --resource-group myResourceGroupAutomate --name myAutomatedVM

PreviousActivate Nitride NextSupported Cloud Configurations

Last updated 15 days ago