In this tutorial, we will create a multi-cloud Kubernetes cluster consisting of 2 nodes (Azure and AWS). The Azure VM will serve as the master node, while the AWS EC2 instance will function as the worker node.
Create a template
The first step will be to create a template for multi-cloud Kubernetes. You can refer to the "Create a dyneemes node template" tutorial for a better understanding of how to do this.
Create a namespace
The second step will be to create a namespace. This is a mandatory requirement for creating attestation. You can learn how to create and use namespaces in the documentation .
Create dkv-v2 engines
The third step is to create engine, where we will store the kubeconfig.
POST
http://localhost:8200/v1/sys/mounts/buckypaper
Headers
Body
Response
Enable K8s PKI
Next, you need to mount the K8s PKI. Below is an example of how to do this using the Vault CLI.
Make sure to add the previously created namespace during this process.
Copy #!/bin/bash
set -x
export VAULT_ADDR = 'http://127.0.0.1:8200'
export KEY_TYPE = 'ec'
export KEY_BITS = '256'
vault secrets enable -namespace=education -path=k8s-pki-root pki
vault secrets enable -namespace=education -path=k8s-pki pki
vault secrets enable -namespace=education -path=k8s-pki-etcd pki
vault secrets enable -namespace=education -path=k8s-pki-front-proxy pki
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-root
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-etcd
vault secrets tune -namespace=education -max-lease-ttl=87600h k8s-pki-front-proxy
vault write -namespace=education -format=json \
k8s-pki-root/config/urls \
issuing_certificates= "${VAULT_ADDR}/v1/k8s-pki-root/ca" \
crl_distribution_points= "${VAULT_ADDR}/v1/k8s-pki-root/crl" \
ocsp_servers= "${VAULT_ADDR}/v1/k8s-pki-root/ocsp"
vault write -namespace=education -format=json \
k8s-pki/config/urls \
enable_templating= true \
issuing_certificates= "${VAULT_ADDR}/v1/k8s-pki/issuer/{{issuer_id}}/der" \
crl_distribution_points= "${VAULT_ADDR}/v1/k8s-pki/issuer/{{issuer_id}}/crl/der" \
ocsp_servers= "${VAULT_ADDR}/v1/k8s-pki/ocsp"
vault write -namespace=education -format=json \
k8s-pki-etcd/config/urls \
enable_templating= true \
issuing_certificates= "${VAULT_ADDR}/v1/k8s-pki-etc/issuer/{{issuer_id}}/der" \
crl_distribution_points= "${VAULT_ADDR}/v1/k8s-pki-etc/issuer/{{issuer_id}}/crl/der" \
ocsp_servers= "${VAULT_ADDR}/v1/k8s-pki-etcd/ocsp"
vault write -namespace=education -format=json \
k8s-pki-front-proxy/config/urls \
enable_templating= true \
issuing_certificates= "${VAULT_ADDR}/v1/k8s-pki-front-proxy/issuer/{{issuer_id}}/der" \
crl_distribution_points= "${VAULT_ADDR}/v1/k8s-pki-front-proxy/issuer/{{issuer_id}}/crl/der" \
ocsp_servers= "${VAULT_ADDR}/v1/k8s-pki-front-proxy/ocsp"
vault write -namespace=education -format=json \
k8s-pki-root/root/generate/internal \
ttl=87600h \
key_type= "${KEY_TYPE}" \
key_bits= "${KEY_BITS}" \
common_name= "Enclaive K8S Root CA"
# main
vault write -namespace=education -format=json \
k8s-pki/issuers/generate/intermediate/internal \
ttl=87600h \
key_type= "${KEY_TYPE}" \
key_bits= "${KEY_BITS}" \
common_name= "Enclaive K8S CA" \
| tee /dev/stderr \
| jq -r '.data.csr' > csr.pem
vault write -namespace=education -format=json \
k8s-pki-root/root/sign-intermediate \
csr=@csr.pem \
format=pem_bundle \
ttl=87600h \
| tee /dev/stderr \
| jq -r '.data.certificate' > cert.pem
vault write -namespace=education -format=json \
k8s-pki/intermediate/set-signed \
certificate=@cert.pem
# etcd
vault write -namespace=education -format=json \
k8s-pki-etcd/issuers/generate/intermediate/internal \
ttl=87600h \
key_type= "${KEY_TYPE}" \
key_bits= "${KEY_BITS}" \
common_name= "Enclaive K8S etcd CA" \
| tee /dev/stderr \
| jq -r '.data.csr' > csr.pem
vault write -namespace=education -format=json \
k8s-pki-root/root/sign-intermediate \
csr=@csr.pem \
format=pem_bundle \
ttl=87600h \
| tee /dev/stderr \
| jq -r '.data.certificate' > cert.pem
vault write -namespace=education -format=json \
k8s-pki-etcd/intermediate/set-signed \
certificate=@cert.pem
# front-proxy
vault write -namespace=education -format=json \
k8s-pki-front-proxy/issuers/generate/intermediate/internal \
ttl=87600h \
key_type= "${KEY_TYPE}" \
key_bits= "${KEY_BITS}" \
common_name= "Enclaive K8S front-proxy CA" \
| tee /dev/stderr \
| jq -r '.data.csr' > csr.pem
vault write -namespace=education -format=json \
k8s-pki-root/root/sign-intermediate \
csr=@csr.pem \
format=pem_bundle \
ttl=87600h \
| tee /dev/stderr \
| jq -r '.data.certificate' > cert.pem
vault write -namespace=education -format=json \
k8s-pki-front-proxy/intermediate/set-signed \
certificate=@cert.pem
Register new workload
POST
http://localhost:8200/v1/auth/ratls/attestation/create
For more detailed information on creating attestation, you can refer to the documentation .
Body
Name Type Description f05d8808-547a-4e9a-9843-07c3f55b7e67
http://localhost:3000/webhook
Headers
Response
200
Copy {
"request_id" : "0bde6eee-f55b-e9a5-e1ba-7a382c6a0d50" ,
"lease_id" : "" ,
"renewable" : false ,
"lease_duration" : 0 ,
"data" : {
"instance" : "77255d88-754c-42a3-954f-58fb86bf48a5"
} ,
"wrap_info" : null ,
"warnings" : null ,
"auth" : null
}
Create nodes
In this step, you must create nodes with a configuration that supports confidential VMs . Before creating the node, you must add cloud-init, an example of which is shown below.
Azure AWS
DC2as_v5 Ubuntu 20_04-lts-cvm
ENV Description Value The provider name that we specified during the measurement creation.
The "instance" field that we obtained during the attestation creation.
77255d88-754c-42a3-954f-58fb86bf48a4
k8s-control or k8s-worker
Cloud Init
Copy #cloud-config
runcmd :
- |
(
set -eu
apt-get update
apt-get upgrade -y
apt-get install -y docker.io curl gnupg2 systemd git openssh-server
systemctl enable --now docker
systemctl enable --now ssh
mkdir -p /etc/apt/keyrings
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" \
| tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key \
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
apt-get update
apt-get upgrade -y
apt-get install -y kubeadm kubelet kubectl
export ENCLAIVE_PROTOCOL=sev-snp
export ENCLAIVE_SOURCE=azure
export ENCLAIVE_INSTANCE=77255d88-754c-42a3-954f-58fb86bf48a4
export ENCLAIVE_RESOURCE=azure-node
export ENCLAIVE_NITRIDE=http://localhost:8200
export ENCLAIVE_KEYSTORE=http://localhost:8200
export ENCLAIVE_DNS_NAME=$(curl -s ifconfig.me)
ENCLAIVE_FEATURES=k8s-control
if [ -x "$(command -v curl)" ];then
COMMAND="wget -q -O"
elif [ -v "$(command -v wget)" ];then
COMMAND="curl -s -o"
else
echo "Not installed: curl|wget"
exit 1
fi
$COMMAND client "$ENCLAIVE_NITRIDE/static/enclaivelet"
$COMMAND provision "$ENCLAIVE_NITRIDE/static/provision"
chmod +x client provision
./client
echo 'PermitRootLogin=yes' | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart sshd
M6a Ubuntu 23.04
ENV Description Value The provider name that we specified during the measurement creation.
The "instance" field that we obtained during the attestation creation.
77255d88-754c-42a3-954f-58fb86bf48a4
k8s-control or k8s-worker
Cloud Init
Copy #cloud-config
runcmd :
- |
(
set -eu
apt-get update
apt-get upgrade -y
apt-get install -y docker.io curl gnupg2 systemd git openssh-server
systemctl enable --now docker
systemctl enable --now ssh
mkdir -p /etc/apt/keyrings
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" \
| tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key \
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
apt-get update
apt-get upgrade -y
apt-get install -y kubeadm kubelet kubectl
mkdir -p /etc/containerd/
containerd config default \
| sed 's/SystemdCgroup \= false/SystemdCgroup \= true/g' > /etc/containerd/config.toml
sudo systemctl restart containerd
export ENCLAIVE_PROTOCOL=sev-snp
export ENCLAIVE_SOURCE=aws
export ENCLAIVE_INSTANCE=77255d88-754c-42a3-954f-58fb86bf48a4
export ENCLAIVE_RESOURCE=aws-node
export ENCLAIVE_NITRIDE=http://localhost:8200
export ENCLAIVE_KEYSTORE=http://localhost:8200
export ENCLAIVE_DNS_NAME=$(curl -s ifconfig.me)
ENCLAIVE_FEATURES=k8s-worker
if [ -x "$(command -v curl)" ];then
COMMAND="wget -q -O"
elif [ -v "$(command -v wget)" ];then
COMMAND="curl -s -o"
else
echo "Not installed: curl|wget"
exit 1
fi
sudo apt-get update -y
sudo apt-get install -y linux-modules-extra-$(uname -r)
sudo modprobe sev-guest
$COMMAND client "$ENCLAIVE_NITRIDE/static/enclaivelet"
$COMMAND provision "$ENCLAIVE_NITRIDE/static/provision"
chmod +x client provision
./client
echo "PermitRootLogin yes" | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart sshd
Once all the steps have been completed, the result of the attestation will be sent to the webhook you specified when creating the attestation. Below is an example of what is sent to the webhook. Ensure that the webhook accepts the HTTP POST method.
Copy {
"Success" : true ,
"Message" : "success" ,
"Instance" : "77255d88-754c-42a3-954f-58fb86bf48a5" ,
"Resource" : "azure-node" ,
"Quote" : "eyJWZXJzaWlE9PSJ9fQ=="
}
Kubeconfig
After completing all the steps, the generated kubeconfig will be located in Vault at the path /:instance/k8s/admin
POST
http://localhost:8200/v1/buckypaper/data/:instance/k8s/admin
Params
Name Value 77255d88-754c-42a3-954f-58fb86bf48a4
Headers
Response
200 400
Copy {
"request_id" : "dbdcb2de-7e0a-36f6-0b16-96a3bbef837b" ,
"lease_id" : "" ,
"renewable" : false ,
"lease_duration" : 0 ,
"data" : {
"data" : {
"value": "apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMxVENDQW51Z0F3SUJBZ0lVS2RTV3VRR3hmK1RORlpzWi96RDdsOFlibmZjd0NnWUlLb1pJemowRUF3SXcKQURBZUZ3MHlOREEwTVRZeE5EQTVNRFZhRncweU9UQTBNVFV4TkRBNU16VmFNRG94T0RBMkJnTlZCQU1UTHpRMQpaVE13TVdaaExUZ3hNVEl0TkRGa09TMWlPREV4TFdVME1qZGpOek0yWldGbFlpQnJPSE10Y0d0cElFTkJNRmt3CkV3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFMUhCOGJTMks3Q3NrOGlnWGRaRFF0em9MQm9CRURkRG0KOXZhSy84bTVwSSthTmFrZWZxRjBXOHVvdm9ybFArSUJFSElISzdFRmlZNzBZWmpBZDh6c3ZLT0NBWmN3Z2dHVApNQTRHQTFVZER3RUIvd1FFQXdJQkJqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUjFXaHJyCnFOL3dTRjVrMW9mdVh4eWxOT2ttK1RBZkJnTlZIU01FR0RBV2dCVDIzcTJtOEpLL01YTnlwdjFscTVyRnp5V0cKSVRDQnV3WUlLd1lCQlFVSEFRRUVnYTR3Z2Fzd1BnWUlLd1lCQlFVSE1BR0dNbWgwZEhCek9pOHZibWwwY21sawpaUzVrWlhZdWJUUnlaMlJ0ZEhJdU1XUXVjSFF2ZGpFdmF6aHpMWEJyYVM5dlkzTndNR2tHQ0NzR0FRVUZCekFDCmhsMW9kSFJ3Y3pvdkwyNXBkSEpwWkdVdVpHVjJMbTAwY21ka2JYUnlMakZrTG5CMEwzWXhMMnM0Y3kxd2Eya3YKYVhOemRXVnlMelV5TlRJNU1HRTFMV001TlRFdE9UZzNZeTB3WTJFNExUSmhOVEk1WW1SbE5qRTVNeTlrWlhJdwpjZ1lEVlIwZkJHc3dhVEJub0dXZ1k0WmhhSFIwY0hNNkx5OXVhWFJ5YVdSbExtUmxkaTV0TkhKblpHMTBjaTR4ClpDNXdkQzkyTVM5ck9ITXRjR3RwTDJsemMzVmxjaTgxTWpVeU9UQmhOUzFqT1RVeExUazROMk10TUdOaE9DMHkKWVRVeU9XSmtaVFl4T1RNdlkzSnNMMlJsY2pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCa092Uk9QQ3crWkxzUQpleFNzbm1uVlJXS2ppY3RpTjNiNlBrVDdXTkYwQXdJaEFMS21XSG9vT2tobUltZUcwNW9YdkVjUk40b3NabEE2CktITGR6djBRNGFIcgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n server: https://185.112.181.197:6443\n name: kubernetes\ncontexts:\n- context:\n cluster: kubernetes\n user: kubernetes-admin\n name: kubernetes-admin@kubernetes\ncurrent-context: kubernetes-admin@kubernetes\nkind: Config\npreferences: {}\nusers:\n- name: kubernetes-admin\n user:\n client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNoakNDQWl1Z0F3SUJBZ0lJU3VFdTdVbG9idFl3Q2dZSUtvWkl6ajBFQXdJd09qRTRNRFlHQTFVRUF4TXYKTkRWbE16QXhabUV0T0RFeE1pMDBNV1E1TFdJNE1URXRaVFF5TjJNM016WmxZV1ZpSUdzNGN5MXdhMmtnUTBFdwpIaGNOTWpRd05ERTJNVFF3T1RBMVdoY05NalV3TkRFMk1UUXhNREk1V2pBME1SY3dGUVlEVlFRS0V3NXplWE4wClpXMDZiV0Z6ZEdWeWN6RVpNQmNHQTFVRUF4TVFhM1ZpWlhKdVpYUmxjeTFoWkcxcGJqQ0NBU0l3RFFZSktvWkkKaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFMWHJRKzNuSEdJTUFHRXp6TFN6aDdmRmc4K1JwRmNDdnBZVAp6MThURkVqUWhXS1pWVVRVQ0xUZWFSN1VsUEJlNHJuOGN2NHp0NURJN0hkRWJIYXNGbW13SG15UFZqbiszQ0ppCjQ5U1pMVXFKNnVQOFEzYmozUUZ4eHpER3B6WmdFQW5XOTJ3d3BoZGI2eklGbVNnOERTcHFqZTA5QlNCak5YV2MKNjkxZVBuWTVTT001ekx2STlGMjd0M0NXdzEycm1FNllZWkNoNTN1Um9rdDVHcmpERElTTWp0S2E5Zy9keGhxego0eXA2Y1VHVlZNUVowdGJnN2o5cE5TaHE1eGRPQUhnK09IMnpVWkY5QlZCM3pzN2hKY2s4VFlLa0dFb2YwT0hVCm9wVzdWQWdHZlhDcCs3a1hlbzNuV0FnWkM3dk5ORlJYMXpkcnY0ZjlBTlJFTDBsR2w1MENBd0VBQWFOV01GUXcKRGdZRFZSMFBBUUgvQkFRREFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQwpNQUF3SHdZRFZSMGpCQmd3Rm9BVWRWb2E2NmpmOEVoZVpOYUg3bDhjcFRUcEp2a3dDZ1lJS29aSXpqMEVBd0lEClNRQXdSZ0loQU1FdG4wV1JGaUsybnNwOHB3bWVmZGJCd2I4WkNaMW04ZHhLWDdibFZMcHZBaUVBeDhlUDBCYUMKVjJRMEk0eUJIU2ttZVJjcklwTkpJblptdHc3NGlYeWtvZ0E9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\n client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBdGV0RDdlY2NZZ3dBWVRQTXRMT0h0OFdEejVHa1Z3SytsaFBQWHhNVVNOQ0ZZcGxWClJOUUl0TjVwSHRTVThGN2l1Znh5L2pPM2tNanNkMFJzZHF3V2FiQWViSTlXT2Y3Y0ltTGoxSmt0U29ucTQveEQKZHVQZEFYSEhNTWFuTm1BUUNkYjNiRENtRjF2ck1nV1pLRHdOS21xTjdUMEZJR00xZFp6cjNWNCtkamxJNHpuTQp1OGowWGJ1M2NKYkRYYXVZVHBoaGtLSG5lNUdpUzNrYXVNTU1oSXlPMHByMkQ5M0dHclBqS25weFFaVlV4Qm5TCjF1RHVQMmsxS0dybkYwNEFlRDQ0ZmJOUmtYMEZVSGZPenVFbHlUeE5ncVFZU2gvUTRkU2lsYnRVQ0FaOWNLbjcKdVJkNmplZFlDQmtMdTgwMFZGZlhOMnUvaC8wQTFFUXZTVWFYblFJREFRQUJBb0lCQUdWdjFNWFA2MXlrY29YQQp1M0U2OWY4N3JEN09hQU40YlVzRHFzckp1YkxNU3NQcTJjZnlMeFNqTzV4TVR1d2xER2xHWWR4cWUvM0llMG9aCnBoMFo0YmwyMGRBWXNLelA5bkZhRU0zWHg1QmJqTlVwTVhrV240SVJyazc5UmZtazRPeUxxQlQwbjNoQThjbEgKbzlueWVpamZsMW5rZjNwS3QyRm9hWUJhNzVzOWJzVS9pMm1jSEdnTFFsNVVyWDUzUkNrNVFTbXFQQTE5TmN1cQpDcit6STFBUWxLbGR0aG9GYlRtWEkxalcrMmF6WEJvZ2x3SlV6QmNSdFpBRVBwRDV3TS9SYXBVaVQ4bkhWQm5uClVZWVpuYjQwVXpBQkl4SWhtaWNuYS9TUXdydjdLNTZsTFJsN2pmcFBITEV1ZituY3lWSEY2RUJHLzBjQ2pmUWUKdkttYWdtRUNnWUVBNWhTQ2EzYmRtSVZMVS9yQXZqUlhWejZKUWpUdFYvMTZwTGlPTEZzS2NONjJnVHI2N1c1bgo0Q0RYaDF3c3g4MFc5ZFI4dE9FVmx5Nzk4c0lqZjNHZDhVZ3VKODR2QUJTeGx2NWh0MDNQRFJpZmZ5enN0N0Z6CmEwbHA5SEJZZDBmbnc0aGprSFBFaWt0MUFxamZaZTRBU0NBRDRnTXhlUlJ2aFVFNmRqRm1oSWtDZ1lFQXltbkwKTGdKZ3cwN01ZWkphMkRvOTcvcHY5b3VKSklyV2Z4UFRkNHh0WHBLcHVYdzdNYTF2cDU3VVdFWjQ5RExJdzFkMQpKbWI1d3lHT1F0dFhYYjg2TTRISlpEVXE0VjFjUXlwY3kzYlNIQndjVVAyMytPWmZoR0VIRTNrUHE2aDZNUzBqCjJWUW9MSTV2SGpPNFU1UVZyOXpTUkE1WEo1N1AvaCtGTmFRRW5YVUNnWUFjT2RmbUp5YjRWeUh1bzR5UnlRb08KQWZZRTBVbHN5Z1FKd1RwdjVwdFpUWGxHVVY0SFVzd2JpdXZtQjBEWElpbjI3cUo5N2o4ZjFQd0NsQjZXdS9HTQpjRFo1VzZvWWNIRjQvS3d3T0xOMS85ZFJXc21QbjQxeFZBaFRuSlM5eld4NWFIb09kcVZkOGZQZzU2YkFjbnBHClh1Y2JDeGsrcVdzbEJlT09MSEZQUVFLQmdCOHR3cVZZcW0wTFEwSTRXQlA4akxQZGdNZWFZTXFuTkRrbzVhY0IKazR0QXpqSUxKOWNVSHlIVHZtUEduelVHYVpSbGNWOHo5MzhPT1NxbFBNVHRBdHNTUCtKV3FqOUNzVWFMVFBYdQpYSmtGMzNxK1NrdGx1UXJjSTBubG1QdFpIVkZiNGF6RllOYlVMZHVhSGVlSjVQbE02M3FlTnVDY2Z1OW5EWTdnCnIvRk5Bb0dBUXdLdmo5RUIxUzczVmJ5MVFyNnFnRDRmL3lVY21MNjlyQ0ZWMkJNNTJvMVVJZDd3THl4b0tJVnkKM2wycWxOMWNSMHBJS0h4dk1sS1BtSmFIMktTL3I4cG9CK3V6TG55NkcvdklMSGlQVU8yR3hRcHZoK2FadTJSbwpsbmtiRng5TTIyZU50Q3NBTXdQY0ZJVFoxZ2kraVhhUUxEVVdqTHVOS2lzS0s3WEMzMDQ9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==\n"
} ,
"metadata" : {
"created_time" : "2024-04-16T14:10:40.804167672Z" ,
"custom_metadata" : null ,
"deletion_time" : "" ,
"destroyed" : false ,
"version" : 1
}
} ,
"wrap_info" : null ,
"warnings" : null ,
"auth" : null
}
Copy {
"error" : "Invalid request"
}
Last updated 6 months ago