Home
Documentation Tutorials

Provisioning kubeconfig

In this tutorial, we will create a multi-cloud Kubernetes cluster consisting of 2 nodes (Azure and AWS). The Azure VM will serve as the master node, while the AWS EC2 instance will function as the worker node.

Create a template

The first step will be to create a template for multi-cloud Kubernetes. You can refer to the "Create a dyneemes node template" tutorial for a better understanding of how to do this.

Create a namespace

The second step will be to create a namespace. This is a mandatory requirement for creating attestation. You can learn how to create and use namespaces in the documentation.

Create dkv-v2 engines

The third step is to create engine, where we will store the kubeconfig.

POST http://localhost:8200/v1/sys/mounts/buckypaper

Headers

NameValue

X-Vault-Token

<token>

X-Vault-Namespace

education

Body

NameTypeValue

type

string

dkv-v2

Response

204 No Content

Enable K8s PKI

Next, you need to mount the K8s PKI. Below is an example of how to do this using the Vault CLI.

Make sure to add the previously created namespace during this process.

#!/bin/bash

set -x

export VAULT_ADDR='http://127.0.0.1:8200'
export KEY_TYPE='ec'
export KEY_BITS='256'

vault secrets enable -namespace=education -path=k8s-pki-root pki
vault secrets enable -namespace=education -path=k8s-pki pki
vault secrets enable -namespace=education -path=k8s-pki-etcd pki
vault secrets enable -namespace=education -path=k8s-pki-front-proxy pki

vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-root
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-etcd
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-front-proxy

vault write -namespace=education -format=json \
  k8s-pki-root/config/urls \
  issuing_certificates="${VAULT_ADDR}/v1/k8s-pki-root/ca" \
  crl_distribution_points="${VAULT_ADDR}/v1/k8s-pki-root/crl" \
  ocsp_servers="${VAULT_ADDR}/v1/k8s-pki-root/ocsp"

vault write -namespace=education -format=json \
  k8s-pki/config/urls \
  enable_templating=true \
  issuing_certificates="${VAULT_ADDR}/v1/k8s-pki/issuer/{{issuer_id}}/der" \
  crl_distribution_points="${VAULT_ADDR}/v1/k8s-pki/issuer/{{issuer_id}}/crl/der" \
  ocsp_servers="${VAULT_ADDR}/v1/k8s-pki/ocsp"

vault write -namespace=education -format=json \
  k8s-pki-etcd/config/urls \
  enable_templating=true \
  issuing_certificates="${VAULT_ADDR}/v1/k8s-pki-etc/issuer/{{issuer_id}}/der" \
  crl_distribution_points="${VAULT_ADDR}/v1/k8s-pki-etc/issuer/{{issuer_id}}/crl/der" \
  ocsp_servers="${VAULT_ADDR}/v1/k8s-pki-etcd/ocsp"

vault write -namespace=education -format=json \
  k8s-pki-front-proxy/config/urls \
  enable_templating=true \
  issuing_certificates="${VAULT_ADDR}/v1/k8s-pki-front-proxy/issuer/{{issuer_id}}/der" \
  crl_distribution_points="${VAULT_ADDR}/v1/k8s-pki-front-proxy/issuer/{{issuer_id}}/crl/der" \
  ocsp_servers="${VAULT_ADDR}/v1/k8s-pki-front-proxy/ocsp"

vault write -namespace=education -format=json \
  k8s-pki-root/root/generate/internal \
  ttl=87600h \
  key_type="${KEY_TYPE}" \
  key_bits="${KEY_BITS}" \
  common_name="Enclaive K8S Root CA"

# main

vault write -namespace=education -format=json \
  k8s-pki/issuers/generate/intermediate/internal \
  ttl=87600h \
  key_type="${KEY_TYPE}" \
  key_bits="${KEY_BITS}" \
  common_name="Enclaive K8S CA" \
  | tee /dev/stderr \
  | jq -r '.data.csr' > csr.pem

vault write -namespace=education -format=json \
  k8s-pki-root/root/sign-intermediate \
  csr=@csr.pem \
  format=pem_bundle \
  ttl=87600h \
  | tee /dev/stderr \
  | jq -r '.data.certificate' > cert.pem

vault write -namespace=education -format=json \
  k8s-pki/intermediate/set-signed \
  certificate=@cert.pem

# etcd

vault write -namespace=education -format=json \
  k8s-pki-etcd/issuers/generate/intermediate/internal \
  ttl=87600h \
  key_type="${KEY_TYPE}" \
  key_bits="${KEY_BITS}" \
  common_name="Enclaive K8S etcd CA" \
  | tee /dev/stderr \
  | jq -r '.data.csr' > csr.pem

vault write -namespace=education -format=json \
  k8s-pki-root/root/sign-intermediate \
  csr=@csr.pem \
  format=pem_bundle \
  ttl=87600h \
  | tee /dev/stderr \
  | jq -r '.data.certificate' > cert.pem

vault write -namespace=education -format=json \
  k8s-pki-etcd/intermediate/set-signed \
  certificate=@cert.pem

# front-proxy

vault write -namespace=education -format=json \
  k8s-pki-front-proxy/issuers/generate/intermediate/internal \
  ttl=87600h \
  key_type="${KEY_TYPE}" \
  key_bits="${KEY_BITS}" \
  common_name="Enclaive K8S front-proxy CA" \
  | tee /dev/stderr \
  | jq -r '.data.csr' > csr.pem

vault write -namespace=education -format=json \
  k8s-pki-root/root/sign-intermediate \
  csr=@csr.pem \
  format=pem_bundle \
  ttl=87600h \
  | tee /dev/stderr \
  | jq -r '.data.certificate' > cert.pem

vault write -namespace=education -format=json \
  k8s-pki-front-proxy/intermediate/set-signed \
  certificate=@cert.pem

Register new workload

POST http://localhost:8200/v1/auth/ratls/attestation/create

For more detailed information on creating attestation, you can refer to the documentation.

Body

NameTypeDescription

template

string

f05d8808-547a-4e9a-9843-07c3f55b7e67

namespace

string

education

webhook

string

http://localhost:3000/webhook

Headers

NameValue

X-Vault-Token

<token>

Response

{
    "request_id": "0bde6eee-f55b-e9a5-e1ba-7a382c6a0d50",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "instance": "77255d88-754c-42a3-954f-58fb86bf48a5"
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Create nodes

In this step, you must create nodes with a configuration that supports confidential VMs. Before creating the node, you must add cloud-init, an example of which is shown below.

DC2as_v5 Ubuntu 20_04-lts-cvm

ENVDescriptionValue

ENCLAIVE_PROTOCOL

sev-snp

ENCLAIVE_SOURCE

The provider name that we specified during the measurement creation.

azure

ENCLAIVE_INSTANCE

The "instance" field that we obtained during the attestation creation.

77255d88-754c-42a3-954f-58fb86bf48a4

ENCLAIVE_RESOURCE

Node name

azure-node

ENCLAIVE_NITRIDE

Nitride URL

http://localhost:8200

ENCLAIVE_KEYSTORE

Vault URL

http://localhost:8200

ENCLAIVE_DNS

IP Address

13.74.123.67

ENCLAIVE_FEATURES

k8s-control or k8s-worker

k8s-control

Cloud Init

#cloud-config
runcmd:
  - |
    (
    set -eu
    
    apt-get update
    apt-get upgrade -y
    apt-get install -y docker.io curl gnupg2 systemd git openssh-server
    systemctl enable --now docker
    systemctl enable --now ssh
    mkdir -p /etc/apt/keyrings
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" \
    | tee /etc/apt/sources.list.d/kubernetes.list
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key \
    | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    apt-get update
    apt-get upgrade -y
    apt-get install -y kubeadm kubelet kubectl

    export ENCLAIVE_PROTOCOL=sev-snp
    export ENCLAIVE_SOURCE=azure
    export ENCLAIVE_INSTANCE=77255d88-754c-42a3-954f-58fb86bf48a4
    export ENCLAIVE_RESOURCE=azure-node
    export ENCLAIVE_NITRIDE=http://localhost:8200
    export ENCLAIVE_KEYSTORE=http://localhost:8200
    export ENCLAIVE_DNS_NAME=$(curl -s ifconfig.me)
    ENCLAIVE_FEATURES=k8s-control

    if [ -x "$(command -v curl)" ];then
      COMMAND="wget -q -O"
    elif [ -v "$(command -v wget)" ];then
      COMMAND="curl -s -o"
    else
      echo "Not installed: curl|wget"
      exit 1
    fi

    $COMMAND client "$ENCLAIVE_NITRIDE/static/enclaivelet"
    $COMMAND provision "$ENCLAIVE_NITRIDE/static/provision"

    chmod +x client provision
    ./client
    
    echo 'PermitRootLogin=yes' | sudo tee -a /etc/ssh/sshd_config
    sudo systemctl restart sshd

Once all the steps have been completed, the result of the attestation will be sent to the webhook you specified when creating the attestation. Below is an example of what is sent to the webhook. Ensure that the webhook accepts the HTTP POST method.

{
  "Success": true,
  "Message": "success",
  "Instance": "77255d88-754c-42a3-954f-58fb86bf48a5",
  "Resource": "azure-node",
  "Quote": "eyJWZXJzaWlE9PSJ9fQ=="
}

Kubeconfig

After completing all the steps, the generated kubeconfig will be located in Vault at the path /:instance/k8s/admin

POST http://localhost:8200/v1/buckypaper/data/:instance/k8s/admin

Params

NameValue

instance

77255d88-754c-42a3-954f-58fb86bf48a4

Headers

NameValue

X-Vault-Token

<token>

X-Vault-Namespace

education

Response

{
    "request_id": "dbdcb2de-7e0a-36f6-0b16-96a3bbef837b",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "data": {
            "value": "apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMxVENDQW51Z0F3SUJBZ0lVS2RTV3VRR3hmK1RORlpzWi96RDdsOFlibmZjd0NnWUlLb1pJemowRUF3SXcKQURBZUZ3MHlOREEwTVRZeE5EQTVNRFZhRncweU9UQTBNVFV4TkRBNU16VmFNRG94T0RBMkJnTlZCQU1UTHpRMQpaVE13TVdaaExUZ3hNVEl0TkRGa09TMWlPREV4TFdVME1qZGpOek0yWldGbFlpQnJPSE10Y0d0cElFTkJNRmt3CkV3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFMUhCOGJTMks3Q3NrOGlnWGRaRFF0em9MQm9CRURkRG0KOXZhSy84bTVwSSthTmFrZWZxRjBXOHVvdm9ybFArSUJFSElISzdFRmlZNzBZWmpBZDh6c3ZLT0NBWmN3Z2dHVApNQTRHQTFVZER3RUIvd1FFQXdJQkJqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUjFXaHJyCnFOL3dTRjVrMW9mdVh4eWxOT2ttK1RBZkJnTlZIU01FR0RBV2dCVDIzcTJtOEpLL01YTnlwdjFscTVyRnp5V0cKSVRDQnV3WUlLd1lCQlFVSEFRRUVnYTR3Z2Fzd1BnWUlLd1lCQlFVSE1BR0dNbWgwZEhCek9pOHZibWwwY21sawpaUzVrWlhZdWJUUnlaMlJ0ZEhJdU1XUXVjSFF2ZGpFdmF6aHpMWEJyYVM5dlkzTndNR2tHQ0NzR0FRVUZCekFDCmhsMW9kSFJ3Y3pvdkwyNXBkSEpwWkdVdVpHVjJMbTAwY21ka2JYUnlMakZrTG5CMEwzWXhMMnM0Y3kxd2Eya3YKYVhOemRXVnlMelV5TlRJNU1HRTFMV001TlRFdE9UZzNZeTB3WTJFNExUSmhOVEk1WW1SbE5qRTVNeTlrWlhJdwpjZ1lEVlIwZkJHc3dhVEJub0dXZ1k0WmhhSFIwY0hNNkx5OXVhWFJ5YVdSbExtUmxkaTV0TkhKblpHMTBjaTR4ClpDNXdkQzkyTVM5ck9ITXRjR3RwTDJsemMzVmxjaTgxTWpVeU9UQmhOUzFqT1RVeExUazROMk10TUdOaE9DMHkKWVRVeU9XSmtaVFl4T1RNdlkzSnNMMlJsY2pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCa092Uk9QQ3crWkxzUQpleFNzbm1uVlJXS2ppY3RpTjNiNlBrVDdXTkYwQXdJaEFMS21XSG9vT2tobUltZUcwNW9YdkVjUk40b3NabEE2CktITGR6djBRNGFIcgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n    server: https://185.112.181.197:6443\n  name: kubernetes\ncontexts:\n- context:\n    cluster: kubernetes\n    user: kubernetes-admin\n  name: kubernetes-admin@kubernetes\ncurrent-context: kubernetes-admin@kubernetes\nkind: Config\npreferences: {}\nusers:\n- name: kubernetes-admin\n  user:\n    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNoakNDQWl1Z0F3SUJBZ0lJU3VFdTdVbG9idFl3Q2dZSUtvWkl6ajBFQXdJd09qRTRNRFlHQTFVRUF4TXYKTkRWbE16QXhabUV0T0RFeE1pMDBNV1E1TFdJNE1URXRaVFF5TjJNM016WmxZV1ZpSUdzNGN5MXdhMmtnUTBFdwpIaGNOTWpRd05ERTJNVFF3T1RBMVdoY05NalV3TkRFMk1UUXhNREk1V2pBME1SY3dGUVlEVlFRS0V3NXplWE4wClpXMDZiV0Z6ZEdWeWN6RVpNQmNHQTFVRUF4TVFhM1ZpWlhKdVpYUmxjeTFoWkcxcGJqQ0NBU0l3RFFZSktvWkkKaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFMWHJRKzNuSEdJTUFHRXp6TFN6aDdmRmc4K1JwRmNDdnBZVAp6MThURkVqUWhXS1pWVVRVQ0xUZWFSN1VsUEJlNHJuOGN2NHp0NURJN0hkRWJIYXNGbW13SG15UFZqbiszQ0ppCjQ5U1pMVXFKNnVQOFEzYmozUUZ4eHpER3B6WmdFQW5XOTJ3d3BoZGI2eklGbVNnOERTcHFqZTA5QlNCak5YV2MKNjkxZVBuWTVTT001ekx2STlGMjd0M0NXdzEycm1FNllZWkNoNTN1Um9rdDVHcmpERElTTWp0S2E5Zy9keGhxego0eXA2Y1VHVlZNUVowdGJnN2o5cE5TaHE1eGRPQUhnK09IMnpVWkY5QlZCM3pzN2hKY2s4VFlLa0dFb2YwT0hVCm9wVzdWQWdHZlhDcCs3a1hlbzNuV0FnWkM3dk5ORlJYMXpkcnY0ZjlBTlJFTDBsR2w1MENBd0VBQWFOV01GUXcKRGdZRFZSMFBBUUgvQkFRREFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQwpNQUF3SHdZRFZSMGpCQmd3Rm9BVWRWb2E2NmpmOEVoZVpOYUg3bDhjcFRUcEp2a3dDZ1lJS29aSXpqMEVBd0lEClNRQXdSZ0loQU1FdG4wV1JGaUsybnNwOHB3bWVmZGJCd2I4WkNaMW04ZHhLWDdibFZMcHZBaUVBeDhlUDBCYUMKVjJRMEk0eUJIU2ttZVJjcklwTkpJblptdHc3NGlYeWtvZ0E9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\n    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBdGV0RDdlY2NZZ3dBWVRQTXRMT0h0OFdEejVHa1Z3SytsaFBQWHhNVVNOQ0ZZcGxWClJOUUl0TjVwSHRTVThGN2l1Znh5L2pPM2tNanNkMFJzZHF3V2FiQWViSTlXT2Y3Y0ltTGoxSmt0U29ucTQveEQKZHVQZEFYSEhNTWFuTm1BUUNkYjNiRENtRjF2ck1nV1pLRHdOS21xTjdUMEZJR00xZFp6cjNWNCtkamxJNHpuTQp1OGowWGJ1M2NKYkRYYXVZVHBoaGtLSG5lNUdpUzNrYXVNTU1oSXlPMHByMkQ5M0dHclBqS25weFFaVlV4Qm5TCjF1RHVQMmsxS0dybkYwNEFlRDQ0ZmJOUmtYMEZVSGZPenVFbHlUeE5ncVFZU2gvUTRkU2lsYnRVQ0FaOWNLbjcKdVJkNmplZFlDQmtMdTgwMFZGZlhOMnUvaC8wQTFFUXZTVWFYblFJREFRQUJBb0lCQUdWdjFNWFA2MXlrY29YQQp1M0U2OWY4N3JEN09hQU40YlVzRHFzckp1YkxNU3NQcTJjZnlMeFNqTzV4TVR1d2xER2xHWWR4cWUvM0llMG9aCnBoMFo0YmwyMGRBWXNLelA5bkZhRU0zWHg1QmJqTlVwTVhrV240SVJyazc5UmZtazRPeUxxQlQwbjNoQThjbEgKbzlueWVpamZsMW5rZjNwS3QyRm9hWUJhNzVzOWJzVS9pMm1jSEdnTFFsNVVyWDUzUkNrNVFTbXFQQTE5TmN1cQpDcit6STFBUWxLbGR0aG9GYlRtWEkxalcrMmF6WEJvZ2x3SlV6QmNSdFpBRVBwRDV3TS9SYXBVaVQ4bkhWQm5uClVZWVpuYjQwVXpBQkl4SWhtaWNuYS9TUXdydjdLNTZsTFJsN2pmcFBITEV1ZituY3lWSEY2RUJHLzBjQ2pmUWUKdkttYWdtRUNnWUVBNWhTQ2EzYmRtSVZMVS9yQXZqUlhWejZKUWpUdFYvMTZwTGlPTEZzS2NONjJnVHI2N1c1bgo0Q0RYaDF3c3g4MFc5ZFI4dE9FVmx5Nzk4c0lqZjNHZDhVZ3VKODR2QUJTeGx2NWh0MDNQRFJpZmZ5enN0N0Z6CmEwbHA5SEJZZDBmbnc0aGprSFBFaWt0MUFxamZaZTRBU0NBRDRnTXhlUlJ2aFVFNmRqRm1oSWtDZ1lFQXltbkwKTGdKZ3cwN01ZWkphMkRvOTcvcHY5b3VKSklyV2Z4UFRkNHh0WHBLcHVYdzdNYTF2cDU3VVdFWjQ5RExJdzFkMQpKbWI1d3lHT1F0dFhYYjg2TTRISlpEVXE0VjFjUXlwY3kzYlNIQndjVVAyMytPWmZoR0VIRTNrUHE2aDZNUzBqCjJWUW9MSTV2SGpPNFU1UVZyOXpTUkE1WEo1N1AvaCtGTmFRRW5YVUNnWUFjT2RmbUp5YjRWeUh1bzR5UnlRb08KQWZZRTBVbHN5Z1FKd1RwdjVwdFpUWGxHVVY0SFVzd2JpdXZtQjBEWElpbjI3cUo5N2o4ZjFQd0NsQjZXdS9HTQpjRFo1VzZvWWNIRjQvS3d3T0xOMS85ZFJXc21QbjQxeFZBaFRuSlM5eld4NWFIb09kcVZkOGZQZzU2YkFjbnBHClh1Y2JDeGsrcVdzbEJlT09MSEZQUVFLQmdCOHR3cVZZcW0wTFEwSTRXQlA4akxQZGdNZWFZTXFuTkRrbzVhY0IKazR0QXpqSUxKOWNVSHlIVHZtUEduelVHYVpSbGNWOHo5MzhPT1NxbFBNVHRBdHNTUCtKV3FqOUNzVWFMVFBYdQpYSmtGMzNxK1NrdGx1UXJjSTBubG1QdFpIVkZiNGF6RllOYlVMZHVhSGVlSjVQbE02M3FlTnVDY2Z1OW5EWTdnCnIvRk5Bb0dBUXdLdmo5RUIxUzczVmJ5MVFyNnFnRDRmL3lVY21MNjlyQ0ZWMkJNNTJvMVVJZDd3THl4b0tJVnkKM2wycWxOMWNSMHBJS0h4dk1sS1BtSmFIMktTL3I4cG9CK3V6TG55NkcvdklMSGlQVU8yR3hRcHZoK2FadTJSbwpsbmtiRng5TTIyZU50Q3NBTXdQY0ZJVFoxZ2kraVhhUUxEVVdqTHVOS2lzS0s3WEMzMDQ9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==\n"
        },
        "metadata": {
            "created_time": "2024-04-16T14:10:40.804167672Z",
            "custom_metadata": null,
            "deletion_time": "",
            "destroyed": false,
            "version": 1
        }
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Last updated