| Memory Encryption: Confidential computing leverages hardware features to encrypt memory within secure enclaves. This ensures that data remains encrypted even while it's being processed, preventing unauthorized access to sensitive information. | | | | Memory Encryption |
| Workload Attestation: Workload attestation verifies the integrity and identity of a workload running within an enclave. It allows external parties to verify that a particular workload is running securely within a trusted execution environment (TEE), ensuring the workload has not been compromised. | | | | Workload Attestation |
| Confidential Boot: Confidential boot ensures the integrity and security of the system during the boot process. It involves establishing a secure chain of trust, verifying the authenticity and integrity of each component in the boot sequence, and protecting against tampering or malicious modifications. | | | | Confidential Boot |
| Sealing/Binding: Sealing or binding refers to encrypting data and tying it to a specific enclave or platform. This ensures that the encrypted data can only be accessed by authorized enclaves or identities, providing confidentiality and protecting against unauthorized access. | | | | Sealing / Binding |
| Secret Provisioning: Provisioning ensures that the enclaves are correctly initialized with the required secrets to process data and workloads securely. | | | | Secret Provisioning |
Enclaive's Confidential Kubernetes cluster has been meticulously designed to simplify the deployment of confidential workloads while ensuring straightforward operation and management. Our primary objective was to incorporate existing tools, best practices, and the workflows DevOps professionals and engineers utilize in their day-to-day operations while minimizing the complexities associated with hardware secure enclaves. The fundamental components of this solution are as follows:
**Normal application stack (left) vs. libOSed application stack (right)**
The decision on how much functionality to include within the enclave depends on the chosen libOS. At one extreme, a library OS can pull most of the application-supporting code of the OS into the enclave. At the other extreme, thin "shim" layers wrap an API layer like the system call table. Pulling more code into the enclave increases the Trusted Computing Base (TCB) size but reduces the complexity and attack surface between the enclave and the untrusted OS. Performance implications depend on reducing enclave border crossings and managing the size of the Enclave Page Cache.
The adversaries' primary objective is to obtain confidential data or disrupt the execution of an enclave. It is important to note that SGX does not guarantee availability since adversaries have control over all computational resources and can launch Denial of Service (DoS) attacks.
# Versions
**Note:** SGXv1, including EPID attestation, is deprecated. Available is SGXv2 and DCAP attestation.
### SGX 1.0
SGXv1 significantly impacted secure computing in 2015 with the introduction of 6th Generation Core processors. It allows developers to partition their applications securely and protect specific code within hardware-encrypted memory enclaves, safeguarding them from unauthorized access and cold boot attacks. To ensure enhanced protection, any unauthorized attempts on an enclave trigger exceptions, enabling immediate detection. To support these capabilities, 18 new instructions were added to the Instruction Set Architecture (ISA) to facilitate secure computation in untrustworthy environments while maintaining security through trusted firmware and microcode implementations.
SGXv1's 18 ISA instructions:
| Supervisor Instruction | Description | User Instruction | Description |
| ---------------------- | --------------------------------- | ---------------- | ----------------------------- |
| ENCLS\[EADD] | Add a page | ENCLU\[EENTER] | Enter an Enclave |
| ENCLS\[EBLOCK] | Block an EPC page | ENCLU\[EEXIT] | Exit an Enclave |
| ENCLS\[ECREATE] | Create an enclave | ENCLU\[EGETKEY] | Create a cryptographic key |
| ENCLS\[EDBGRD] | Read data by debugger | ENCLU\[EREPORT] | Create a cryptographic report |
| ENCLS\[EDBGWR] | Write data by debugger | ENCLU\[ERESUME] | Re-enter an Enclave |
| ENCLS\[EEXTEND] | Extend EPC page measurement | | |
| ENCLS\[EINIT] | Initialize an enclave | | |
| ENCLS\[ELDB] | Load an EPC page as blocked | | |
| ENCLS\[ELDU] | Load an EPC page as unblocked | | |
| ENCLS\[EPA] | Add version array | | |
| ENCLS\[EREMOVE] | Remove a page from EPC | | |
| ENCLS\[ETRACK] | Activate EBLOCK checks | | |
| ENCLS\[EWB] | Write back/invalidate an EPC page | | |
In SGXv1, the enclave size is limited to 128MB, which is primarily suited to desktop applications. It was primarily focused on applications such as digital rights management and key management. One successful implementation was the end-to-end streaming security in the Blu-ray standard. However, due to limited commercial adoption, Intel discontinued its desktop SGX line and shifted its strategy towards server CPUs, which led to the introduction of SGXv2.
### SGX 2.0
Intel SGXv2 is found in the Icelake and Sapphire dedicated server CPU series. It introduced six additional instructions to the ISA. The main change in SGXv2 was the support for significantly larger and dynamically growing enclave sizes, reaching up to 1 TB (2\*512 GB on a 2-socket board) to accommodate cloud-based applications. Another notable change addressed the remote attestation framework. While privacy was a crucial aspect in DRM applications, the concept of attesting a "streaming" enclave was based on an enhanced privacy identity (EPID) that protected the desktop user's privacy. With SGXv2, Intel revamped the attestation framework to meet the requirements of cloud service providers, resulting in the Data Center Attestation Primitive (DCAP).
SGXv2's 6 ISA instuctions:
| Supervisor Instruction | Description | User Instruction | Description |
| ---------------------- | ------------------------------------ | ------------------- | --------------------------- |
| ENCLS\[EAUG] | Allocate page to an existing enclave | ENCLU\[EACCEPT] | Accept changes to a page |
| ENCLS\[EMODPR] | Restrict page permissions | ENCLU\[EMODPE] | Enhance access rights |
| ENCLS\[EMODT] | Make page TCS | ENCLU\[EACCEPTCOPY] | Copy page to a new location |
Other features that could be considered part of SGXv2 are:
The EPCM ensures memory protection and prevents virtual address translations from compromising enclave security. For example, if the system software attempts to allocate the same EPC page to two enclaves, the corresponding SGX instruction for allocation will fail.
There are four data structures associated with enclave operation within the EPC:
#### Derivation Mechanism
The key derivation process involves two stages. Each update to the processor logic is signed and assigned a Security Version Number (SVN), representing the security revision of the platform. By employing a pseudorandom function (PRF) with the Security Version Number and the Root Provisioning Key as inputs, a "versioned" key is derived, bound to the current platform version. This "versioned" key, along with other inputs, is then used to derive the keys mentioned above (refer to the Figure above). The parameters used in this process include:
.png?alt=media)
{% endhint %}
{% hint style="info" %}
**Note:** Remove `nodeName` selector or replace it to the selector that can select SGX available nodes correctly.
.png?alt=media)
{% endhint %}
2. The `pccs/pccs.yaml` YAML file requires an API key from the Intel PCS API. Add this API key to the environment variable `APIKEY` in the YAML file.
.png?alt=media)
.png?alt=media)
.png?alt=media)
.png?alt=media)
The primary goal of the security architecture is to protect the TDX Linux guest OS from attacks originating from the hypervisor through communication interfaces. Additionally, the goal is to avoid introducing new attack vectors that could compromise the TDX Linux guest kernel (ring 0) from the TDX guest userspace (ring 3). The threat model does not directly cover threats arising from the TDX guest userspace utilizing the mentioned interfaces exposed to an untrusted host/VMM. For instance, if the TDX guest userspace enables debug or test tools that perform MMIO or PCI config space reading without properly validating untrusted host/VMM input, it may introduce additional attack opportunities.
The threat model assumes the hypervisor running the protected TDX guest, thus excluding other hypervisors and their hardening measures from its scope. Also, the threat model doesn't address potential abuses of kernel printout and debug routines that may accept parameters directly from an untrusted host/VMM.
It's important to note that while the TDX module and the Intel platform contribute to the protection of TDX guest memory and registers, they cannot defend the TDX guest from host/VMM attacks that leverage existing communication interfaces between them. These include TDVMCALL hyper calls through the TDX module and shared memory for I/O. To ensure complete security, extra measures, such as end-to-end communication security, must be implemented to prevent the injection of malicious code into the confidential VM.
# Concepts
# Architecture
## Prerequisites
TDX comprises two main components:
### I/O Model
The TDX threat model considers hypervisors and peripheral devices untrusted, prohibiting direct access to the private memory of TDs. Therefore, TDs and their owners must secure I/O data before it crosses the trust boundary. This involves sealing I/O data buffers and placing them in shared memory, identified by the shared bit in the Guest's Physical Address. Hypervisors and peripheral devices can move data to and from the shared memory. Consequently, the guest kernel requires modifications to support this I/O model. Additionally, all I/O data entering TDs from hypervisors or peripheral devices must undergo thorough examination and validation, as it can no longer be considered trustworthy.
In Linux guest support for TDX, all MMIO (Memory Mapped I/O) regions and DMA (Direct Memory Access) buffers are mapped as shared memory within TDs. The Linux guest enforces using SWIOTLB (Software I/O Translation Lookaside Buffer) to allocate and convert DMA buffers to unified locations. This protects against malicious inputs from I/O, and only a limited number of hardened drivers are permitted within TDs.
# TDX Module
In preparation for the loading process of the Intel TDX Module it needs to be ensured that the platform has been configured correctly. As the platform configuration is the job of the BIOS/UEFI, which is untrusted in TDX and SGX Intel developed a piece of firmware named MCHECK which checks these parameters during the initialization process. The parameters which are checked by MCHECK are for example the configuration of the Convertible Memory Ranges, the configuration of memory encryption or the configuration of the ECC memory. The MCHECK routine is embedded into the microcode of the CPU and therefore signed and encrypted by Intel.
The loading process commences with the execution of the Intel Non-Persistent SEAM Loader (NP-SEAM Loader), an Intel Authenticated Code Module (ACM). ACMs are Intel-signed modules that operate within the processor's internal RAM. The NP-SEAM Loader is authenticated and loaded by the Intel Trusted Execution Technology (TXT) via the GETSEC\[ENTERACCS] function.
Inside the NP-SEAM Loader resides the image of the Intel Persistent SEAM Loader (P-SEAM Loader), which is verified and loaded by the NP-SEAM Loader itself. The responsibility of installing or updating the TDX Module lies with the P-SEAM Loader. I want to emphasize that the P-SEAM Loader and the TDX Module are loaded into the SEAM RANGE, a dedicated portion of system memory reserved through UEFI/BIOS. The IA32\_SEAMRR\_PHYS\_BASE and IA32\_SEAMRR\_PHYS\_MASK MSRs specify this range's base address and size. The SEAM RANGE is divided into Module\_Range for the TDX Module and P\_SEAMLDR\_Range for the P-SEAM Loader. Both modules operate in the SEAM VMX root mode and utilize SEAMCALL / SEAMRET instructions to interact with external software.
The NP-SEAM Loader, P-SEAM Loader, and TDX Module are all provided and signed by Intel, establishing a chain of trust to bootstrap the TDX Module. The P-SEAM Loader offers a SEAMCALL interface function named seamldr\_install for loading the TDX Module. The TDX Module's image is pre-loaded into a memory buffer (located outside the SEAM RANGE). The buffer's physical addresses and a seam\_sigstruct (signature of the TDX Module) are passed as parameters to seamldr\_install. The seam\_sigstruct contains the hash value and the Security Version Number (SVN) of the TDX Module, as well as the counts of per-Logical Processor (LP) stack pages, per-LP data pages, and global data pages. These values are used by seamldr\_install to determine the physical/linear addresses and sizes of various memory regions in the TDX Module.
seamldr\_install must be called on all Logical Processors (LPs) serially. When seamldr\_install is invoked on the first LP, an installation session commences. On each subsequent LP, seamldr\_install checks that the LP is not already in an installation session (initiated by another LP) and clears the LP's VMCS (Virtual Machine Control Structure) cache. Upon calling seamldr\_install on the last LP, the following actions are performed:
1. Check the parameters provided to seamldr\_install.
2. Verify the signature of the TDX Module.
3. Check the SVN of the to-be-loaded image and compare it with the resident TDX Module.
4. Determine the physical and linear addresses and sizes of various memory regions within the SEAM RANGE, including code, data, stack, page table, sysinfo\_table, keyhole, and keyhole-edit.
5. Map the physical addresses of these regions to their corresponding linear addresses.
6. Load the binary image of the TDX Module into the SEAM RANGE, measuring the image and computing/verifying the TDX Module's hash value.
7. Set up the TDX Module's sysinfo\_table.
8. Establish SEAM Transfer VMCS on each LP.
9. Record the TDX Module's hash and SVN in the P-SEAM Loader's data region.
In addition to the SEAMCALL used for installing the TDX Module, the P-SEAM Loader provides other interface functions to shut itself down and retrieve the loader's system information.
# Memory Encryption
# Confidentiality and Integrity
Intel TDX promises to secure the data of a Trust Domain against the hypervisor and other VMs running on the same Hardware. In order to successfully fulfil this promise Intel TDX needs an confidentiality and integrity protection mechanism. This mechanism needs to ensure that the content of the memory pages is secured so that other VMs or the hypervisor can not read or manipulate it.
Therefore Intel TDX uses Multi-Key-Total-Memory-Encryption which is a CPU enabled feature that encrypts the memory. The memory controller on an MKTME enabled CPU is extended with an encryption engine that provides the NIST approved AES-128-XTS and AES-256-XTS encryption algorithms. The encryption engine itself holds all encryption keys in the Key Encryption Table (KET). This table maps the encryption keys from a specific TD to its memory pages which are identified through the Host Key Identifier (HKID). With the help of this table the memory controller can encrypt and decrypt memory on the fly.
Intel TDX also offers integrity protection for the private memory pages of TDs in order to protect the TDs against further attacks. Intel TDX supports two modes of operation for its integrity protection, the Cryptographic Integrity (CI) and Logical Integrity (LI) mode. The Logical Integrity mode protects the encrypted memory through the usage of a TD Owner bit while the Cryptographic Integrity mode also adds a Message Authentication Code (MAC). The TD Owner bit is used to indicate that the data that was written is associated with a private HKID. The MAC in CI mode is computed for each cache line of 64 Bytes and also includes an encryption tweak which is the encrypted physical address for the current cache line. The MAC is also computed over the TD Owner bit. The MAC computation uses SHA-3-256 \[1] as an underlying function and the MAC is shorted to 28 bit instead of 256 bit \[2, p. 120].
The mentioned values TD Owner bit and MAC are used to identify poisoned memory upon reading it. Depending on the protection mode (CI vs LI) the CPU reacts differently. However once the CPU realizes that the memory it is trying to read has been poisoned, it writes an indication back that is associated with the memory address back into the memory. This allows the CPU on subsequent reads to detect that the memory has already been detected as poisoned and no further integrity checks will be performed. This indication can be cleared if a new write on that address is issued and the whole memory line is being written. In addition to reduce the attack vector any read on private TD data that comes from an operation outside of the SEAM mode returns all 0. This prevents the testing of MAC collisions with malicious ciphertext.
The following graphic shows what happens when a memory read occurs in CI Mode. We will decide between what happens when a page is meant to be private vs when a page is meant to be shared.
If a page is meant to be private, which is indicated through its HKID, a check of the TD Owner Bit is performed first. If the CPU concludes that the TD Owner Bit is set to zero a mismatch is recognized as the page was previously written using a shared HKID. Therefore the CPU returns zeroed data and indicates that the page has now to be treated as poisoned. If the TD Owner Bit however is set to zero and the MAC comparison is successful the decrypted data is returned. A failure with zeroed data and a poison marker for the page is returned, when the MAC comparison is not successful or the CPU notices that this page belongs to another TD.
A reverse logic is applied to shared memory pages. If the TD Owner Bit is set to one a TD bit mismatch failure is detected, as the page is marked private. The TD Owner Bit zero correctly indicates that a shared memory page is being used. If the MAC check is successful the page is being decrypted and no poison entry is being added. However if the Integrity check fails the CPU marks the page as poisoned and returns zeroed data.
If the TDX Module operates in LI Mode only a TD Owner Bit mismatch for private pages leads to a new poison. Otherwise when the TD Owner Bit mismatch happens for shared pages zeroed data is returned but no poison indication. Instead the the memory controller logs an error which can cause a shutdown or machine check on a subsequent read. If the TD Owner Bit matches the memory controller behaves the same way as in CI mode.
\[1] 
As the figure above, the attestation process involves several steps to ensure the authenticity and integrity of the TD on the TDX platform.
1. The remote party initiates an attestation request, providing a nonce for freshness and protection against replay attacks.
2. The attestation agent within the TDX platform retrieves a TD report from the TDX Module, incorporating the provided nonce as REPORTDATA.
3. The attestation agent then sends the TD report to the Quoting Enclave (QE) to request the attestation key's signature. Before signing the TD report, the QE verifies the platform's legitimacy to ensure the report's origin.
4. The attestation agent receives the signed quote from the QE and forwards it to the remote party.
5. The remote party needs the platform's Provisioning Certification Key (PCK) certificate to validate the quote. It may obtain the PCK certificate from a Provisioning Certification Caching Service (PCCS) (Step 5) or directly from the Intel Provisioning Certification Service (PCS) (Step 6).
6. The remote party performs several checks during the validation process. It verifies the presence of the nonce in the quote and ensures the integrity of the signature chain from the Intel-issued PCK certificate to the signed quote. It also examines the certificate chain for revocation and validates the TCB's currency. Additionally, the remote party verifies whether the measurements in the quote, such as MRTD and RTMR values, match a predefined set of reference values.
7. Upon successfully validating the quote, the remote party can trust that the TD has been securely instantiated on a genuine TDX platform.
# Platform Registration
## Overview
A multi-package platform refers to a computer system built with multiple processors or processing units (CPUs) working together. These individual CPUs are often separate physical packages that can be added or removed from the system.
### Registration Modes
Intel’s registration authority service supports two mechanisms for registering a platform.
#### Direct Registration
* You send your platform details directly to Intel's registration service.
* This grants them permission to store encrypted copies of your platform keys.
* They can then generate a Provisioning Certificate Key (PCK) quickly using just your platform ID.
* **Downside:** You lose control over your platform keys (they're stored by Intel).
#### Indirect Registration
* You don't send your platform details directly to Intel.
* Instead, you create a Provisioning Certificate Key (PCK) yourself using your platform details.
* Intel's service only uses your details temporarily to create the PCK, then discards them.
* They only store basic platform information, not the keys themselves.
* **Downside:** You need to keep a copy of your platform details to generate future PCKs.
### Registration Environments
#### Single-Stage Registration
* A registration agent is running automatically at boot time in order to directly speak with the Registration Authority Service via the Internet.
* The registration agent can directly access the stored values in the BIOS and contact the Registration Authority Service at boot time for platform verification.
* **Downside:** The platform requires access to the Internet.
#### Dual-Stage Registration
* The environment where the platform is placed has no direct access to the internet.
* Instead a proxy server is running between the agent and the authority service.
* **Downside:** The proxy server only supports platform manifest retrieval. The Add Package flow requires a direct response from the Registration Authority Service and is therefore only feasible if an immediate response is available from the Registration Authority Service or if the platform provisioning can be completed with the Platform Membership Certificate later.
# Attestation Report
Enabling remote attestation for a Trusted Domain (TD) involves several steps. Firstly, measurements of the loaded software are taken during both the build-time and runtime of the TD. Subsequently, the TD's measurements and platform Trusted Computing Base (TCB) information are retrieved, generating a TD report. Finally, a quote is derived from the TD report, which can be used by a third party to verify the authenticity of the TD and ensure it operates on a legitimate TDX platform with the expected TCB versions and software measurements.
| TDX Measurement Registers | TPM PCRs | Usage |
| ------------------------- | ---------- | ------------------------------------ |
| MRTD | PCR\[0] | Virtual firmware |
| RTMR\[0] | PCR\[1,7] | Virtual firmware data + configure |
| RTMR\[1] | PCR\[2-5] | OS kernel + Initrd + boot parameters |
| RTMR\[2] | PCR\[8-15] | OS application |
| RTMR\[3] | N/A | Reserved |
### Taking Measurements
Taking measurements in TDX involves using two types of measurement registers for each Trusted Domain (TD): The Measurement of the Trust Domain (MRTD) for build-time measurements and four Runtime Measurement Registers (RTMRs) for runtime measurements. These registers can be likened to the TPM's PCRs (Platform Configuration Registers).
The MRTD captures measurements during the TD build process. When the hypervisor adds initial memory pages to the TD during its creation, the MRTD in the TDCS (Trust Domain Control Structure) is extended with measurements of these pages. To initiate measurement, the hypervisor uses the SEAMCALL\[TDH.MEM.PAGE.ADD] instruction, calculating a SHA384 update over the ASCII string "MEM.PAGE.ADD" and the Guest Physical Address (GPA) of the page. The resulting hash value extends the MRTD. After copying the page into TD's memory (i.e., mapped and available in the SEPT - Secure Enclave Page Table), the hypervisor calls SEAMCALL\[TDH.MR.EXTEND] multiple times to measure the page's content in 256 B blocks. For each block, another SHA384 update is calculated over the ASCII string "MR.EXTEND" and the GPA of the block. A second SHA384 update is also performed over the block's content. Both hash values contribute to extending the MRTD. These initial pages typically contain TD's virtual firmware. The MRTD's measurement excludes pages containing control structures (TDR, TDCS, and TDVPS) and the SEPT. Once the initial set of pages is added, the hypervisor finalizes the MRTD measurement using SEAMCALL\[TDH.MR.FINALIZE], preventing further extensions to the MRTD. For example, when initializing a TD using KVM as a Linux hypervisor, the TDVF (virtual firmware of a TD) code is measured into the MRTD.
The RTMRs are general measurement registers labelled 0 through 3, serving TD's runtime measurements. A TD can employ these registers to achieve a measured boot, measuring all software loaded after booting. The RTMRs start initialized to zero. To extend the content of an RTMR, the TD calls TDCALL\[TDG.MR.RTMR.EXTEND] with the index of the measurement register and a 64 B-aligned physical address of the 48 B extension buffer containing the value. This call calculates a SHA384 hash over the current value of the specified index measurement register concatenated with the value in the extension buffer: RTMR\[index] = SHA384(RTMR\[index] || value). For instance, TDVF measures the static/dynamic configuration data into RTMR\[0] and the OS kernel, boot parameters, and initrd into RTMR\[1].
### Generating a TD Report
Inside a Trusted Domain (TD), a TD report is generated using the TDX Module's report function, invoked by the TD calling TDCALL\[TDG.MR.REPORT]. This call takes a newly initialized report structure and user report data named REPORTDATA, which is 64 B in size and serves as a nonce to ensure the TD report's freshness. To fulfil the request, the TDX Module utilizes the newly introduced SEAMOPS\[SEAMREPORT] instruction, incorporating TD's measurements and REPORTDATA. The CPU then appends Trusted Computing Base (TCB) information related to the SEAM and generates the TD report, protected with integrity, using an HMAC key exclusively accessible to the CPU. Subsequently, the TDX Module returns this TD report to the TD. However, I want to point out that the TD report's verification can only be done on the same platform using the EVERIFYREPORT2 instruction and cannot be performed off the platform.
The figure below illustrates the TD report's structure, comprising three components: REPORTMACSTRUCT, TEE\_TCB\_INFO, and TD\_INFO. The REPORTMACSTRUCT contains header information specifying the structure type as TDX and includes fields for the CPU's Security Version Number (SVN) and hashes of the TEE\_TCB\_INFO and TD\_INFO components. Additionally, it incorporates the REPORTDATA provided as input to the TDCALL\[TDG.MR.REPORT] function. Lastly, the entire header is protected by an HMAC, ensuring the integrity of the TEE\_TCB\_INFO and TD\_INFO components. The TEE\_TCB\_INFO structure contains details regarding the SVN and measurements of the TDX Module. In contrast, the TD\_INFO has TD attestable properties, such as the initial TD configuration and values of the measurement registers.
### Deriving a Quote
The TD report needs to be converted into a quote to enable verification by a third party outside the platform. TDX leverages the remote attestation mechanism of SGX for this purpose. The TD initiates a call to request the Quoting Enclave (QE), which operates on the host platform, to sign the TD report. Depending on the platform's quoting service implementation, this call can be made through VSOCK or TDCALL\[TDG.VP.VMCALL].
After receiving the request, the QE uses the EVERIFYREPORT2 instruction to verify the HMAC protection of the TD report. If the verification succeeds, the QE uses its certified attestation key to sign the TD report, creating a quote digitally. This operation effectively replaces the MAC integrity protection of the TD report with digital signature protection. Consequently, any party can verify the authenticity and integrity of the quote using public key certificates. This quote can then be used for remote attestation, enabling external entities to verify the legitimacy of the TD and the platform's Trusted Computing Base (TCB) versions and software measurements.
# AMD SEV
AMD Secure Encrypted Virtualization (SEV) represents a CPU capability that encompasses the subsequent attributes:
The SEV-SNP threat model goes beyond the scope of previous AMD SEV technologies, addressing additional attack vectors and potential threats to VM security. SEV and SEV-ES used a threat model of a "benign but vulnerable" hypervisor, implying that the hypervisor was not completely secure but was trusted to act with benign intent. SEV and SEV-ES technologies helped limit the exposure of certain hypervisor bugs or raise the difficulty of exploitation.
In contrast, SEV-SNP considers the hypervisor and other components as fully untrusted, aiming to protect against integrity attacks such as data replay, corruption, re-mapping, and aliasing-based attacks. Availability is also ensured, guaranteeing that the hypervisor retains control of the system and can regain control or terminate a guest VM at any time.
Confidentiality: In all current SEV technologies, confidentiality threats are addressed through hardware-based memory encryption. This ensures that an untrusted component, such as the hypervisor or a DMA-capable device, cannot directly access the plaintext data inside a VM. However, in cases where the VM explicitly allows untrusted access to a page, exceptions may apply. SEV-ES technology also adds confidentiality protection for the VM's register state by encrypting it when the VM returns to the hypervisor. This protection is maintained in SEV-SNP as well.
The AES encryption key used with this mechanism, known as Secure Memory Encryption (SME), is randomly generated during each system reset and remains concealed from any software running on the CPU cores. Management of the encryption key is entirely entrusted to the AMD Secure Processor (AMD-SP), a 32-bit microcontroller (ARM® Cortex®-A5) serving as a dedicated security subsystem integrated within the AMD SOC. The key is generated using the onboard NIST SP 800-90 compliant hardware random number generator and securely stored in dedicated hardware registers within the SOC, ensuring that it is never exposed outside the system in plaintext. Unlike the SEV mode (to be discussed later), SME does not necessitate the participation of software running on the CPU cores in key management.
The decision of which memory pages should be encrypted is controlled by the operating system (OS) or hypervisor (HV) through the software-managed page tables. When memory encryption is enabled, the 47th bit of the physical address (referred to as the C-bit, as for enCrypted) is utilized to indicate whether a memory page should be protected. The OS or HV sets the 47th bit of the page table entry (PTE) to 1 to signify that the page should be encrypted. Subsequently, accesses to that memory are automatically encrypted and decrypted by the AES engine in the memory controller.
It is essential to note that the encryption and decryption processes via the AES engine introduce a minor increase in latency for DRAM memory accesses. The impact of this latency on software performance varies significantly depending on the system workload but is generally estimated to have only a marginal effect on overall system performance. If only a subset of memory is encrypted, the performance impact will be less pronounced since unencrypted accesses typically do not incur additional latency, as illustrated in figure above.
The utilization of the C-bit in the page tables offers flexibility for the OS or HV to selectively encrypt only a subset of memory, if desired. This approach still provides physical protection of the encrypted memory, while potentially improving performance for non-sensitive data. Additionally, the choice between encrypted and unencrypted memory can be used to create isolation for critical workloads. One example of this isolation is a system that marks only the memory used by guest VMs as encrypted. This can be achieved without modifying the guest VMs themselves. By setting the C-bit in all the nested page table entries corresponding to DRAM for those VMs (as depicted in figure above), the hypervisor enables encryption for the VM memory exclusively.
This configuration can be employed to protect VMs from a rogue machine administrator. Even if the administrator has access to the hardware and host system, they would not be able to inspect guest VM data, even with memory scanning utilities.
# SEV-SNP Concepts
AES encryption, as employed in AMD SEV technologies, significantly enhances memory confidentiality protection. It ensures that VM data stored in DRAM remains indecipherable to attackers without knowledge of the encryption key. The SEV memory encryption key is generated via a hardware random number generator and stored in dedicated hardware registers, inaccessible to direct software access. Moreover, the encryption mechanism ensures that the same plaintext at different memory locations will result in different ciphertexts.
While encryption provides a strong layer of defense, determined attackers may attempt integrity attacks, altering memory values without knowing the encryption key. Such attacks manipulate the data in memory, causing it to deviate from the VM's intended values. Two common types of integrity attacks are memory corruption leading to random data for the VM and replay attacks, where an attacker captures ciphertext from a previous point in time and later substitutes it back into memory.
Integrity attacks themselves do not directly compromise a VM; they only become potent when software inside the VM relies on the corrupted data, leading to potential compromises or information disclosure. The impact of such attacks depends on how the software behaves when encountering compromised data. Predicting the behavior of software in such situations becomes challenging since it typically remains unaware of its memory integrity being compromised.
SEV-SNP is specifically designed to counter software-based integrity attacks and mitigate the risks associated with compromised memory integrity. **The core principle of SEV-SNP is to ensure that if a VM can read a private (encrypted) memory page, it must always retrieve the last value written to that location.** If it attempts to read the memory and encounters a different value, an exception is raised to indicate that the memory cannot be read. This guarantee remains intact even if the VM undergoes memory page swapping to disk or migration to a new host. Implementing this integrity guarantee requires a combination of new CPU hardware and firmware, which will be further discussed in this white paper.
In standard use cases, VMs not only execute their tasks but also communicate with external entities through I/O operations. This communication often involves shared (unencrypted) memory. Outgoing data that the VM wishes to make available is placed in a shared memory page, and incoming data is similarly stored in a shared page. Since shared memory is not encrypted with the VM's specific key, it is vital to use appropriate software encryption protocols like HTTPS for securing I/O traffic.
In an SEV-SNP VM, the RMP check is more intricate. The virtual address undergoes translation to a Guest Physical Address (GPA) through AMD-V 2-level paging. The GPA is then further translated to a System Physical Address (SPA), which is used to index the RMP. The entry in the RMP is inspected to confirm that the page is owned by the specific guest, mapped at the particular GPA, and contains the expected mapping information. If any of these checks fail, an exception is raised, and the access is denied.
Not all memory accesses require an RMP check. For instance, read accesses from the hypervisor or non-SEV-SNP guests are exempted since data confidentiality is already protected through AES memory encryption. However, all write accesses, regardless of the mode, necessitate an RMP check. This includes standard memory writes and A/D-bit updates during page table walks. The results of RMP checks are cached in the CPU TLB and related structures to improve efficiency.
Due to the critical role of the RMP in enforcing memory access control, it is not directly writable by software. Special CPU instructions are available to enable manipulation of RMP entries, allowing the hypervisor to assign pages to specific guests or reclaim them. When necessary, hardware automatically performs TLB invalidations to ensure that all processors in the system receive updated RMP entry information, maintaining memory security across all cores.
# Page Validation
As mentioned earlier, each entry in the Reverse Map Table (RMP) contains information about the Guest Physical Address (GPA) at which a specific page of DRAM should be mapped. This ensures that each System Physical Address (SPA) can only be mapped to a single GPA at any given time. Conversely, allowing a single GPA to be mapped to multiple SPAs would compromise the integrity guarantee of SEV-SNP. Although nested page tables enforce the one-to-one GPA-to-SPA mapping, the hypervisor retains the ability to modify these tables at will. To ensure the desired integrity, SEV-SNP introduces the concept of Validation.
Within each RMP entry, there is a Validated bit that is automatically cleared to 0 by CPU hardware when a new RMP entry is created for a guest. Pages assigned to guests with the Validated bit cleared cannot be used by the hypervisor or as private guest pages since they are not yet validated. The guest can only use these pages after setting the Validated bit through a new CPU instruction called PVALIDATE. Only the guest has access to this instruction, and each guest VM can only validate its own memory.
The process of adding a new page to a guest VM involves a two-step process, as shown in the figure above. First, the hypervisor assigns the page to the guest using the new RMPUPDATE instruction, which sets the page to the Guest-Invalid state. Second, the guest validates the page using the PVALIDATE instruction, transitioning it to the Guest-Valid state, where it becomes usable.
To achieve the desired integrity, the guest VM should never validate memory corresponding to the same GPA more than once. This can be accomplished in two ways: either the guest VM validates all of its memory at boot and refuses to validate additional memory except as part of hot-plug events, or the guest VM keeps track of validated memory locations and refuses to validate the same location twice. If the guest VM correctly validates its memory, it guarantees an injective mapping between GPAs and SPAs. The guest will validate each GPA only once, and the RMP table ensures that each SPA can only map to one GPA.
By correctly implementing page validation, re-mapping attacks like the one shown in figure above can be blocked. In this scenario, GPA A is initially mapped to SPA X, and the guest validates this translation through PVALIDATE, setting the Validated bit in the corresponding RMP entry. If the hypervisor maliciously attempts to remap A to a different SPA Y, it will create an RMP entry for SPA Y while attempting to map GPA A using the RMPUPDATE instruction. Subsequently, the hypervisor alters the nested page table (NPT) to re-map GPA A to Y. However, when the guest accesses Y, it encounters a #VC (VMM Communication) exception. This exception occurs because the Validated bit in the RMP entry corresponding to SPA Y is clear (since RMPUPDATE initially cleared it). Recognizing that it has already validated GPA A, the guest concludes that it should not be facing a validation error and realizes it is under attack, prompting it to take steps to protect itself, such as terminating or employing other defensive measures
# Page States
As illustrated, the Reverse Map Table (RMP) in SEV-SNP plays a crucial role in tracking the state of each memory page, governing its permissible uses, the entities authorized to read or write it, and potential state transitions. The various page states determine the level of access and control granted to different entities. For instance:
1. Hypervisor state: Pages in this state can be read and written by the hypervisor or SEV-SNP VMs accessing the memory with C=0 (shared pages).
2. Guest-Valid state: Pages in this state can be read and written by SEV-SNP VMs but are protected from being written by the hypervisor.
3. Guest-Invalid state: Pages in this state are yet to be validated by the guest, and access to them is restricted.
The SEV-SNP architecture defines a total of eight main page states, as listed in Table 3. Page state transitions are depicted in figure above and can occur through three mechanisms: the new CPU RMPUPDATE instruction (red), the new PVALIDATE instruction (blue), or via the VM management API in the AMD-SP (green).
Similar to previous SEV technologies, SEV-SNP incorporates a VM management API in the AMD-SP. The hypervisor employs this interface to facilitate VM lifecycle tasks and page management. To maintain security, any pages that the AMD-SP will manipulate must first be placed into special states called Immutable states before initiating the necessary API call. Pages in the Immutable states are protected from being written by any software on the CPU (hypervisor or guest) and cannot have their RMP entry modified by any entity other than the AMD-SP. After processing a page in one of these Immutable states, the AMD-SP can transition it to a different state as dictated by the specific API call.
One such example of Immutable pages is 'Metadata' pages. These pages are writable only by the AMD-SP and serve to store metadata entries associated with guest pages that have been swapped to disk. When a page is swapped to disk, the AMD-SP creates a metadata entry containing an authentication tag (from AES-GCM) and relevant data from the page's RMP entry, such as the GPA where it was located. As the Metadata page itself is not writable by the hypervisor, the integrity of this information is guaranteed. When a page is swapped back into memory, the AMD-SP verifies that the contents remain unchanged and ensures that the page is reintegrated into the guest address space at the same location as before. Metadata pages themselves can also be swapped to disk in a similar manner, enabling the entire guest to be saved to disk if needed.
# Virtual Machine Privilege Levels
Virtual Machine Privilege Levels (VMPLs) represent a new optional feature within the SEV-SNP architecture, offering a means for a guest VM to partition its address space into four hierarchical levels. These levels serve as hardware-isolated abstraction layers, providing enhanced security controls and aiding in communication management with the hypervisor.
The VMPLs are arranged hierarchically, with VMPL0 being the highest privilege level and VMPL3 the least privileged. When enabled, each virtual CPU (vCPU) of a VM is assigned a specific VMPL. The Reverse Map Table (RMP) entry for each page of private guest memory is extended to include page access rights corresponding to each VMPL. These permissions are applied in conjunction with standard paging permissions. Essentially, individual guest pages can be marked as readable, writeable, supervisor-mode executable, or user-mode executable. Initially, when a page is validated by a guest, VMPL0 is granted full permissions for the page, while all other VMPLs are granted no permissions. However, the guest can choose to adjust these VMPL permissions using the new RMPADJUST instruction.
The RMPADJUST instruction allows a given VMPL to modify the permissions for a less privileged VMPL. For example, VMPL0 could grant read and write (but not execute) permissions on a page to VMPL1. This granting of permissions is limited to prevent a higher-level VMPL from granting more permissions than it currently possesses. The primary purpose of VMPLs is to set additional page permission checks, and they do not interfere with other x86 security features.
The RMP page permission checks occur during the RMP lookup at the end of a table-walk. These checks are restrictive, meaning that for a guest page to be writable, it must be marked as writable in the guest-managed page tables (associated with the active vCPU), the nested page tables managed by the hypervisor, and the RMP table managed by a higher privileged VMPL.
VMPLs resemble nested virtualization to some extent since a guest can have its own management layer running at a higher VMPL, controlling permissions on its other pages. This facilitates use cases such as securely virtualizing a security-enforcing hypervisor. In a bare-metal system, a standard hypervisor may be employed to enforce certain page permissions, such as making certain pages read-only or non-executable. SEV-SNP enables this same level of control and protection in a cloud environment. In such a scenario, VMPL0 inside the guest would enforce the required page permissions since the actual hypervisor in the cloud is considered untrusted.
VMPLs offer valuable applications beyond serving as an abstraction layer. For instance, in SEV-SNP, certain VMs might require a more restrictive environment, and therefore, they may choose to move APIC (Advanced Programmable Interrupt Controller) emulation within the guest's trust domain. To achieve this, VMPL0 can be utilized to perform trusted APIC emulation while the rest of the guest operates in lower VMPLs, remaining unaware of the emulation process.
Another use case for VMPL0 is acting as an intermediary for guest-to-hypervisor communications. In previous SEV and SEV-ES technologies, an enlightened guest OS was necessary to handle security features, such as setting the C-bit in page tables, managing #VC (VMM Communication) exceptions (in SEV-ES), and more. However, in SEV-SNP, these tasks can optionally be delegated to VMPL0.
In this approach, VMPL0 can configure which guest memory in another vCPU is private (C=1) versus shared (C=0) using a watermark called the Virtual Top of Memory (vTOM). Memory addresses below vTOM are automatically treated as private, while memory above vTOM is treated as shared. By using vTOM to distinguish memory in this manner, there is no need to modify the standard x86 page tables with C-bit markings, simplifying guest OS software.
Additionally, VMPL0 can handle #VC events that occur in another vCPU. An SEV-SNP VM can be configured such that when an intercepted instruction is executed in one vCPU (e.g., RDMSR - Read Model Specific Register), that vCPU exits, and VMPL0 is invoked. VMPL0 can then access the intercepted information directly from the encrypted save area of the original vCPU, execute any necessary hyper-calls, and emulate the instruction on behalf of the original vCPU. While not as efficient as a natively enlightened guest, this behavior enables VMPL0 to act as the bridge logic for running unenlightened (legacy) guest VMs. This capability has the potential to secure older workloads that may not easily upgrade to newer operating systems.
Overall, VMPLs extend the versatility of SEV-SNP, allowing for a range of security configurations and enabling new approaches for communication, isolation, and emulation within guest VMs.
# Interrupt/Exception Protection
To address potential concerns about interrupt and exception handling in virtual machine (VM) operating systems, SEV-SNP introduces two optional modes that VMs can enable to establish a more restrictive interface with the hypervisor.
1. **Restricted Injection**: In this mode, virtual interrupt queuing is disabled, and the hypervisor's ability to inject interrupts is partially restricted. The hypervisor is only permitted to inject a single newly defined exception vector, #HV, which serves as a doorbell. Restricted Injection assumes that VMs and the hypervisor communicate events in a para-virtualized manner, like using an event queue in shared memory. The #HV exception serves as a signal to the guest VM to re-scan the event queue for new information.
2. **Alternate Injection**: In this mode, standard virtual interrupt queuing and injection interfaces are allowed, but their control is exclusively assigned to the guest itself. Additional fields are added to the encrypted save area, referred to as the Virtual Machine Save Area (VMSA), to facilitate interrupt queuing and event injection. As the VMSA is accessible only by someone with access to the guest VMSA data (such as VMPL0), it ensures that interrupt-related security-sensitive states, like Task Priority Register (TPR), are securely saved and protected from manipulation by a malicious hypervisor.
When used together, these two modes enable VMPL0 to handle interrupt processing and APIC emulation. Virtual CPUs (vCPUs) assigned to run VMPL0 can be operated with Restricted Injection enabled, allowing them to communicate with the hypervisor using a para-virtualized interface and the #HV exception. Conversely, vCPUs utilized for other VMPLs, typically running the main OS of the guest, can be run with Alternate Injection enabled. This configuration allows VMPL0 to safely inject events and virtual interrupts to the main OS. From a software standpoint, the main OS can utilize a standard APIC or x2APIC interface, and any APIC accesses it requires can be trapped and emulated in VMPL0.
In this manner, the VMPL interrupt handling ensures a more secure and controlled environment for VMs and helps prevent potential disruptions caused by malicious hypervisor actions or unwarranted assumptions about interrupt and exception behavior based on bare-metal hardware. The use of different VMPL modes allows for a well-defined separation of concerns and facilitates effective communication between VMs and the hypervisor.
# Trusted Platform Information
Traditionally, platform features and capabilities are discovered using the CPUID instruction. Hypervisors often trap and emulate this instruction for various purposes, such as restricting guest access to certain features to facilitate migration. However, when a malicious hypervisor provides incorrect CPUID information, it can potentially lead to security issues.
For example, if a hypervisor lies about the size of the x86 Extended Save Area (used with the XSAVE/XRSTOR CPU instructions), it may cause a guest to allocate insufficient memory and experience a buffer overflow when executing the hardware XSAVE instruction. Although guest VMs can attempt to validate the CPUID information they receive, this validation process can be challenging, especially during the boot process.
To simplify the validation task for guests, SEV-SNP introduces an optional capability called CPUID filtering, which involves the participation of the AMD Secure Processor (AMD-SP). The AMD-SP verifies that the CPUID results reported by the hypervisor do not exceed the platform's capabilities and ensures the accuracy of security-sensitive information, such as the x86 Extended Save Area size.
CPUID filtering can be performed either on-the-fly or during guest boot. In the on-the-fly filtering approach, after receiving CPUID information, the guest can request verification from the AMD-SP to validate security-sensitive data. Alternatively, during VM launch, a dedicated 'CPUID page' can be created, containing pre-vetted CPUID information from the AMD-SP. This enables the guest to access trusted CPUID information from the early stages of booting. Besides enhancing security, this special page facilitates quicker access to CPUID information, thus accelerating the VM boot process.
# TCB Versioning
Within the SEV-SNP architecture, several firmware components are considered upgradeable and trusted in the threat model. These components, such as the AMD-SP API and CPU microcode patch, collectively form the Trusted Computing Base (TCB) of the architecture. To maintain security and integrity, new versions of these firmware components may be released periodically to address bugs, introduce improvements, or implement new features. In the event of discovering a security bug in one of these components, the guest owner needs assurance that their virtual machine (VM) is running on a patched firmware version and not a vulnerable one.
Previous SEV and SEV-ES features relied on a self-reported AMD-SP version number to manage TCB versioning. In SEV-SNP, this check has been significantly enhanced to provide cryptographic strength. The version numbers of all TCB components are now combined with a fused secret called the Chip Endorsement Key (CEK) to generate a Versioned Chip Endorsement Key (VCEK). The VCEK is a private ECDSA key unique to each AMD chip, running a specific TCB version. The construction of the VCEK leverages cryptographic hash functions to prevent any attempt to impersonate a newer TCB version. This ensures that a given TCB version cannot be misrepresented as a more recent one. The VCEK plays a crucial role in various aspects, including signing attestation reports, contributing to the overall security and integrity of the SEV-SNP ecosystem.
# VM Launch & Attestation
Similar to previous SEV and SEV-ES architectures, SEV-SNP VMs start from an initial unencrypted image that typically includes the guest VM's boot code and other non-sensitive data. During the launch process, the hypervisor requests the AMD-SP to install this initial set of pages in the guest. The AMD-SP measures the contents of these pages cryptographically to generate a launch digest. Additionally, SEV-SNP extends this process to include measuring the metadata associated with these pages, such as the guest physical address (GPA) where the pages are placed and their type, ensuring that the launch digest captures both the layout and contents of the initial guest memory.
In SEV-SNP, after the launch process, the guest owner can provide a signed Identity Block (IDB) to associate with the VM. The IDB contains fields that uniquely identify the VM and includes the expected launch digest. The IDB can only be associated with VMs that match the expected launch digest, and it becomes part of the attestation report.
Unlike SEV and SEV-ES, which support attestation only during guest launch, SEV-SNP provides more flexible attestation capabilities. Attestation reports can be requested from the AMD-SP by the guest VM through a protected path at any time. During SEV-SNP VM launch, a set of private communication keys is created by the AMD-SP, enabling the guest to communicate directly with the AMD-SP. This path allows the guest to request attestation reports, cryptographic keys, and other information as needed.
Attestation reports contain the IDB information from launch, system details, and a block of arbitrary data supplied by the guest VM as part of the report request. The attestation report is signed by the AMD-SP firmware using the VCEK. These reports enable third parties, such as the guest owner, to validate that specific data originated from a particular VM.
For instance, a VM can publish a public key and request an attestation report containing the hash of this public key. A third party can then verify the association between the public key and the VM through the attestation report. The report also serves as proof that the VM operated with the appropriate security features enabled and started on an authentic AMD platform. Since the attestation report is signed by the VCEK, its verification confirms both the authenticity of the platform and the TCB version in use (as the VCEK is derived from the TCB version). Upon successful attestation, the guest owner can decide to provide the VM with secrets, like disk decryption keys or other necessary operation keys.
Furthermore, SEV-SNP supports various use models for generating guest key material. VMs can directly request keys from the AMD-SP for specific purposes, such as data sealing. These keys can be derived from different sources, and the VM can select which sources to use based on their use case. For example, local sealing keys may be requested, which are specific to the current part at a certain TCB level and particular to the IDB signing key. These controls ensure that the VM can request keys that cannot be derived by malicious actors or other devices, enhancing the security of the system.
# VM Migration
In modern cloud architectures, VM migration, especially live VM migration, is a common feature used for load balancing, host system maintenance, and other purposes. All SEV technologies, including SEV-SNP, support VM migration, but SEV-SNP enhances the flexibility associated with this process.
In SEV and SEV-ES, VM migration was governed by a policy supplied by the guest owner. This policy specified whether the VM was migratable and, if so, to which type of systems. The AMD-SP was responsible for enforcing this migration policy by authenticating the AMD-SP on the destination machine before initiating the migration.
In SEV-SNP, the task of migration policy enforcement has been delegated to a new entity known as the Migration Agent (MA). The MA itself is an SEV-SNP VM running on the same physical system as the primary VM. During VM launch, the primary VM can optionally be associated with a running MA, and this information is present in its attestation report as the MA is part of the guest's Trusted Computing Base (TCB). Each VM can only be associated with a single MA, but a single MA can manage the migration of multiple VMs.
The MA is responsible for determining which systems the primary VM can migrate to. While the detailed architecture of the MA is beyond the scope of this document, it can implement complex migration policies using various methods.
In a typical cloud scenario, the MA itself is not migratable. Instead, a separate instance of the MA runs on each physical machine. When a VM is about to migrate, the MA on the source machine authenticates the MA on the destination machine and establishes a protected network connection. Upon successful authentication, the MA transfers the required guest information so that the guest can be reconstituted on the new machine.
It's important to note that the flexibility of the MA allows for scenarios where both the source and destination machines need not be online simultaneously. When a VM is paused, its state can be exported to its associated MA. The MA can then choose to move this state to another MA immediately for live migration or hold onto it or store it for the long term. Later, the VM can be reconstituted on the same machine or another one, as determined by the MA. This approach provides greater control and adaptability in managing VM migration in cloud environments.
# Side Channels
Recently, there has been significant security research on CPU side-channel attacks, which exploit the internal structures of the CPU to leak information. One prominent example is Spectre Variant 2, which leverages hardware branch prediction to create data leakage under specific conditions. AMD has responded to such attacks by introducing hardware capabilities to aid software in defending against them, such as Spectre Variant 2.
Spectre Variant 2 demonstrated that the indirect branch predictor (BTB) could be exploited when certain software conditions are met, assuming an attacker can influence the branch predictions of another entity. In AMD's latest CPU designs, support has been added for the SPEC\_CTRL MSR and PRED\_CMD MSR, which provide more software control over the BTB structure. In the SEV-SNP architecture, the SPEC\_CTRL MSR is virtualized, allowing the guest to set its own speculation policy independently of the hypervisor. This empowers the guest to use modes like IBRS (Indirect Branch Restricted Speculation) for added security.
In traditional virtualization, the hypervisor employs protective measures against guest-based attacks, such as retpoline or running with IBRS enabled. However, when the hypervisor is not trusted, the guest may also be concerned about hypervisor-based attacks, including malicious attempts to poison BTB entries. To address this, SEV-SNP VMs can enable additional protection, wherein the CPU hardware prevents the VM from speculatively using BTB entries installed by other entities. This feature automatically performs a BTB flush when necessary to prevent speculative use of such entries.
Simultaneous Multi-Threading (SMT) is another area of CPU hardware that has been targeted in side channel research due to shared hardware resources. To counter potential observation channels, sensitive SEV-SNP VMs can choose to be restricted to running only on SMT-disabled systems.
While SEV-SNP provides options for protection from certain speculative side channel attacks and SMT concerns, it cannot safeguard against all possible side channel attacks. Attacks like PRIME+PROBE, which target specific vulnerable software algorithms based on cache or TLB access patterns, are not protected by SEV-SNP. It is the responsibility of VM owners to follow standard security practices and update their software to avoid such vulnerabilities.
Other side channel attack categories, like application fingerprinting attacks, are outside the scope of SEV-SNP. These attacks aim to determine the application being run, but they do not directly compromise the confidentiality or integrity of guest VM data. Future versions of SEV may include additional protections against some of these attack vectors.
In summary, while SEV-SNP offers valuable protections against certain side channel attacks, it is crucial for VM owners to stay vigilant, follow best security practices, and keep their software up to date to safeguard against potential vulnerabilities.
# Use Cases
SEV, centered on the concept of hardware VMs, offers enhanced security across various use cases. Leveraging main memory encryption, SEV provides the same security advantages as SME, safeguarding against physical attacks as discussed earlier. Additionally, SEV can be employed to protect environments in the following scenarios:
### Cloud
With the exponential growth of cloud computing, especially Infrastructure as a Service (IaaS) data centers, computational power has become more affordable. However, this growth has brought security challenges, particularly concerning the trustworthiness of cloud infrastructure and personnel. Handling sensitive data like health records or trade secrets raises concerns, and sharing hardware among multiple customers may compromise various workloads. Despite the efforts of software designers, there have been instances where isolation measures have failed, leading to the exposure of sensitive code or data.
SEV addresses these challenges by elevating security in IaaS clouds, providing robust security isolation rooted in the hardware itself. While existing technologies like Microsoft's BitLocker® and LUKS protect data-at-rest on hard drives, SEV goes a step further by protecting data-in-use. This cryptographic protection ensures that customer workloads remain isolated from each other and are shielded from potential threats posed by the hosting software. Even a malicious administrator at a cloud data center would be unable to access data within a hosted VM.
### Sandboxing
SEV utilizes hardware VM constructs to establish secure sandbox environments where software can execute while being protected from all other software on the system. These sandboxes can be as large as a full VM with its own disk and OS, or they can be smaller, providing more fine-grained isolation. For instance, SEV hardware can cryptographically isolate Docker containers from the host system, providing better protection for confidential data.
In both cloud and sandboxing scenarios, SEV brings hardware-based security measures that complement existing software-based protection, creating a more robust and resilient security environment.
# Architecture
SEV, an extension to the AMD-V architecture, facilitates the execution of multiple virtual machines (VMs) controlled by a hypervisor. When SEV is enabled, the hardware tags all code and data with its VM ASID, identifying the originating or intended VM for the data. This tag remains attached to the data within the SOC at all times, ensuring that only the owner VM can access it.
Inside the SOC, the tag provides protection to VM data. However, data outside the SOC is secured by AES with 128-bit encryption. When data enters or exits the SOC, the hardware encrypts or decrypts it based on the associated tag's encryption key.
Each VM and the hypervisor have unique tags and encryption keys. This arrangement, combined with memory encryption, restricts data access solely to the VM associated with the relevant tag. In the event that anyone else, including the hypervisor, attempts to access the data, they will only see it in its encrypted form. This robust encryption-based isolation ensures strong cryptographic separation between VMs and between VMs and the hypervisor, bolstering overall system security.
# Encrypted Memory
SEV leverages the same high-performance memory encryption engine featured in the SME technology described earlier. It also utilizes the C-bit in the page tables to indicate encrypted pages, albeit with some additional restrictions.
One of SEV's key features is granting guest VMs the capability to designate which data memory pages they want to keep private. This control is exercised through the standard CPU page tables, fully managed by the guest. Private memory is encrypted with a guest-specific key, while shared memory may be encrypted using the hypervisor key. This flexibility empowers VMs to mark specific memory pages as confidential (private), while others can be used for communication with other VMs or the hypervisor. In a common configuration, the guest typically designates all its code and data as private, except for specific shared pages it chooses to expose.
For enhanced security, SEV hardware enforces that certain memory types (including instruction pages and page tables) must always remain private to protect the VM.
An illustrative example of communication configuration is presented in figure above. In this scenario, the SEV-enabled guest and the hypervisor communicate through memory marked as shared by both entities. However, all other guest memory is encrypted with the guest's key, which the hypervisor cannot utilize directly. Any memory not used by the hypervisor for direct guest communication is encrypted using the SME feature, as detailed earlier. This setup ensures a secure and controlled communication environment between the guest VM and the hypervisor.
# Key Management
The security of SEV relies heavily on the protection of memory encryption keys. Exposure of these keys to malicious entities, including a malicious or flawed hypervisor, poses a significant risk to SEV-protected guests. While the hypervisor is responsible for managing the guest and its resources, it must never gain access to the memory encryption keys themselves. To achieve this, the SEV firmware running within the AMD Secure Processor (AMD-SP) provides a secure key management interface.
The hypervisor utilizes this interface to enable SEV for secure guests and carry out common hypervisor tasks, such as guest launching, running, snapshotting, migrating, and debugging.
To safeguard SEV-enabled guests, the firmware assists in enforcing three crucial security properties:
1. **Authenticity of the Platform**: This property ensures that the platform is genuinely an authentic AMD platform with SEV capabilities. The platform's identity key, signed by both AMD and the owner of the platform, demonstrates its authenticity and ownership.
2. **Attestation of a Launched Guest**: The firmware provides a signature of various components of the SEV-related guest state, including initial memory contents, to the guest owner for verification. This attestation proves that the guest securely launched with SEV enabled and assures the guest owner that the hypervisor did not tamper with SEV initialization before transmitting confidential information to the guest.
3. **Confidentiality of the Guest's Data**: To achieve this, the SEV firmware encrypts guest memory using a memory encryption key known only to itself. This ensures that the hypervisor is prevented from accessing the keys and, subsequently, the guest's sensitive data.
The secure key management interface guarantees robust protection for SEV-enabled guests, mitigating potential security risks associated with memory encryption keys.
Authenticating the platform serves as a crucial defense against malicious software or rogue devices attempting to impersonate a legitimate platform. The platform's authenticity is verified using its identity key, which is signed by AMD to demonstrate that the platform is indeed an authentic AMD platform with SEV capabilities. Additionally, the owner of the platform signs the identity key to establish ownership and administration of the machine. This information is shared with the owners of guests or other instances of the firmware on remote platforms, ensuring a secure and trusted environment (see figure below).
Attestation of the guest launch provides reassurance to guest owners that their guests have launched securely with SEV enabled. The firmware generates a signature of various components of the SEV-related guest state, including the initial memory contents. This signature is then provided to the guest owner, enabling them to verify that the guest is in the expected state. This attestation ensures that the hypervisor has not tampered with SEV initialization before confidential information is transmitted to the guest.
An illustrative process is shown in Figure above. The guest owner initially provides the guest image to the cloud system, where the SEV firmware assists in launching the guest and sends a measurement back to the guest owner. If the guest owner validates this measurement as correct, they then provide additional secrets (such as a disk decryption key) to the running guest, enabling it to proceed with start-up.
To maintain guest confidentiality, memory is encrypted using a memory encryption key known exclusively to the SEV firmware. The SEV management interface strictly prevents the export of the memory encryption key or any other secret SEV state outside the firmware without proper authentication of the recipient. This ensures that the hypervisor cannot access the keys and, consequently, the guest's sensitive data.
The interface also facilitates secure guest data migration to another SEV-capable platform. During this process, the guest's memory contents remain encrypted during transmission. Once the remote platform is authenticated, the SEV firmware securely sends the guest's memory encryption keys to enable the remote platform to run the guest. This secure transport mechanism allows the hypervisor to implement migration and snapshot functions while maintaining SEV-enabled security.
# Software Implications
### Hypervisor
While SEV still relies on the hypervisor for various virtual machine functions, such as device emulation and scheduling, it significantly reduces the hypervisor's role in ensuring security. A guest operating with SEV enabled can protect itself by designating memory pages as private when they should not be shared beyond the virtual machine.
During runtime, the hypervisor communicates with the AMD-SP (AMD Secure Processor) to coordinate the management of memory encryption keys. This communication occurs through the AMD-SP driver and involves tasks like informing the AMD-SP when a VM is about to be executed, allowing the AMD-SP to load the appropriate encryption key into the AES-128 encryption engine. The hypervisor also interacts with the AMD-SP to establish secure mechanisms for guest attestation, migration, and other operations.
Despite having control over the ASID (Address Space Identifier) used to execute a VM and selecting the encryption key, this is not a security concern because a loaded encryption key is meaningless unless the guest was already encrypted with that key. If an incorrect key is loaded or the wrong ASID is used for a guest, the first instruction fetch of that guest will fail as memory will be decrypted with the wrong key, leading to the execution of invalid data and likely causing a fault.
### Guest
An OS running in an SEV-enabled guest must be aware of this new hardware feature and configure its page tables accordingly. Similar to the SME full memory encryption mode, most DRAM addresses are configured to have the C-bit (bit 47) set to 1.
One essential consideration for an SEV-enabled guest is that the SEV hardware does not allow DMA (Direct Memory Access) into guest encrypted memory due to security reasons. All DMA, whether from actual hardware or a hypervisor emulated device, must occur to shared guest memory. Consequently, the guest OS can either allocate memory pages for DMA as shared (C-bit clear) or copy data to/from a special buffer (known as a "bounce buffer") for DMA purposes. Some operating systems already support bounce buffers, such as the swiotlb functionality in Linux.
Lastly, SEV supports multi-core guests, and data can be shared among these cores with no additional performance penalty. The hypervisor simply needs to use the same ASID for all virtual CPU instances of a particular guest.
# ARM CC
The Armv9 architecture is poised to introduce a novel secure execution environment known as Realms. Realms requires only minimal adjustments to the hardware and heavily relies on firmware for its functionalities. This design allows existing applications to continue functioning without modification, while benefiting from heightened security.
Within this section, we delve into the inner workings of Realms. We explore how the correctness of the firmware was established through the application of the Coq proof assistant. Additionally, we present performance benchmarks that compare the efficiency of Realms and outline key distinctions between Realms and established secure enclaves.
# Technology
**COMING SOON!**
{% hint style="info" %}
The official overview: [ARM Confidential Compute Architecture](https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture)
{% endhint %}
ARM Confidential Compute Architecture (CCA) is an approach to securing sensitive data and computations within the ARM ecosystem. It provides a framework for executing trusted code and protecting data confidentiality, integrity, and privacy, even in the presence of potentially compromised software layers or untrusted environments.
The goal of ARM CCA is to enable secure computing across a range of devices, including edge devices, IoT devices, smartphones, servers, and cloud environments. It leverages ARM's hardware-based security features, such as TrustZone technology, to establish isolated execution environments called secure enclaves. These secure enclaves provide a protected space where sensitive code and data can be executed and stored, shielding them from unauthorized access and tampering.
ARM CCA builds on the foundation of ARM TrustZone, which divides the processor's resources into secure and non-secure worlds. The secure world is isolated from the non-secure world, creating a trusted execution environment (TEE) where critical operations can be performed. ARM CCA extends this concept by introducing additional security features and capabilities to further enhance the protection of sensitive data and computations.
Some key components and features of ARM CCA include:
### Dynamic Granule Access and Protection
Each world in Arm CCA operates within its distinct Physical Address Space (PAS), where memory granules (4 KB frames of physical memory) belong exclusively to a single PAS at any point. Notably, individual granules can transition dynamically from NS PAS to Realm PAS, promoting flexibility and shared resource utilization. A Granule Protection Table (GPT) enforces an access control policy that prevents unauthorized memory accesses, ensuring data security. The GPT supports differentiated memory access rights for each security state, minimizing the risk of data breaches.
