{'fraction_non_alphanumeric': 0.03178383527832507, 'fraction_numerical': 0.0072964254010868855, 'mean_word_length': 4.935997943709035, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Trusted Container Extensions for Container-based Confidential Computing

Cloud computing has emerged as a corner stone of today's computing landscape. More and more customers who outsource their infrastructure benefit from the manageability, scalability and cost saving that come with cloud computing. Those benefits get amplified by the trend towards microservices. Instead of renting and maintaining full VMs, customers increasingly leverage container technologies, which come with a much more lightweight resource footprint while also removing the need to emulate complete systems and their devices. However, privacy concerns hamper many customers from moving to the cloud and leveraging its benefits. Furthermore, regulatory requirements prevent the adaption of cloud computing in many industries, such as health care or finance. Standard software isolation mechanisms have been proven to be insufficient if the host system is not fully trusted, e.g., when the cloud infrastructure gets compromised by malicious third-party actors. Consequently, confidential computing is gaining increasing relevance in the cloud computing field. We present Trusted Container Extensions (TCX), a novel container security architecture, which combines the manageability and agility of standard containers with the strong protection guarantees of hardware-enforced Trusted Execution Environments (TEEs) to enable confidential computing for container workloads. TCX provides significant performance advantages compared to existing approaches while protecting container workloads and the data processed by them. Our implementation, based on AMD Secure Encrypted Virtualization (SEV), ensures integrity and confidentiality of data and services during deployment, and allows secure interaction between protected containers as well as to external entities. Our evaluation shows that our implementation induces a low performance overhead of 5.77% on the standard SPEC2017 benchmark suite.

Abstract-Cloud computing has emerged as a corner stone of today's computing landscape. More and more customers who outsource their infrastructure benefit from the manageability, scalability and cost saving that come with cloud computing. Those benefits get amplified by the trend towards microservices. Instead of renting and maintaining full VMs, customers increasingly leverage container technologies, which come with a much more lightweight resource footprint while also removing the need to emulate complete systems and their devices.
However, privacy concerns hamper many customers from moving to the cloud and leveraging its benefits. Furthermore, regulatory requirements prevent the adaption of cloud computing in many industries, such as health care or finance. Standard software isolation mechanisms have been proven to be insufficient if the host system is not fully trusted, e.g., when the cloud infrastructure gets compromised by malicious third-party actors. Consequently, confidential computing is gaining increasing relevance in the cloud computing field.
We present Trusted Container Extensions (TCX), a novel container security architecture, which combines the manageability and agility of standard containers with the strong protection guarantees of hardware-enforced Trusted Execution Environments (TEEs) to enable confidential computing for container workloads. TCX provides significant performance advantages compared to existing approaches while protecting container workloads and the data processed by them. Our implementation, based on AMD Secure Encrypted Virtualization (SEV), ensures integrity and confidentiality of data and services during deployment, and allows secure interaction between protected containers as well as to external entities. Our evaluation shows that our implementation induces a low performance overhead of 5.77% on the standard SPEC2017 benchmark suite.

I. INTRODUCTION
For over a decade, there is a continuous trend towards cloud computing, which allows customers to leverage capability and cost advantages. Cloud computing evolved with the advent of virtualization [9]. Virtual machines (VMs) enabled Infrastructure-as-a-Service (IaaS) which allows businesses and users to outsource pre-existing workloads to the cloud. However, in recent years, the trend in cloud computing has shifted from VM-based offerings to more lightweight solutions, in particular, container technologies [1], [19], [25], [32].
Containers, such as Docker [19], provide multiple separated user-space instances, which are isolated from each other and the host system through kernel software mechanisms.
By running directly on the host system, containers do not need complex device emulation, large virtual machine disk files and packages pre-configured applications with all their dependencies which makes them an attractive choice for fast deployment of webservices. Cloud providers today recognized this trend and offer customers the possibility to deploy and manage containers in the cloud [1], [25], known as called Container-as-a-Service (CaaS), with Docker being currently the most popular container ecosystem [19], [47].
Despite offering many advantages, using cloud services introduces a risk of data being exposed to third parties or services being compromised [38]. Furthermore, regulatory policies [20] restrict the adoption of cloud services for many industries, such as health care or finance. Even if the cloud service provider (CSP) is considered trustworthy, the CSP's infrastructure might be compromised, e.g., by insiders such as maliciously acting administrators and employees, nation state actors demanding access by law, as well as third-party entities. While the hypervisor software components, which are used to control and manage VMs, have been subject to various attacks 1 , the attack surface in CaaS settings is even larger as a typically large and complex operating system kernel is responsible for managing and isolating the containers 2 .
In recent years, confidential computing has gained relevance in the realm of cloud computing in a pursue to enable the trustworthy outsourcing of sensitive data and services to the cloud, while eliminating the requirement to trust the CSP. Leveraging hardware-enforced Trusted Execution Environments (TEEs), the user's workloads are protected inside isolated compartments, called enclaves, which are secure even if the host's privileged software is compromised or controlled by a malicious entity. Various TEE architectures have been proposed by academic research [8], [15], [16], [18], [21], [34], [37], [48], [49], while commercially available and widely deployed TEEs are Arm TrustZone [5], Intel SGX [28] and AMD SEV [30]. Just Recently, Intel and Arm announced new TEE architectures named Intel Trust Domain Extensions (TDX) [29] and Arm Confidential Compute Architecture (CCA) [6]. However, none of the available TEE architectures is designed to isolate container workloads and to securely orchestrate and manage those. The demand for securely isolated container is shown by efforts to isolate containers in enclaves using Intel SGX [7] or Arm TrustZone [56], however, these approaches either suffer from unpractical performance overheads or cannot protect from malicious cloud providers.
In this paper, we present Trusted Container Extensions (TCX), a novel security architecture providing strongly isolated containers that can be securely deployed and managed in the cloud. We leverage existing TEE architectures, such as AMD SEV, Intel TDX or Arm CCA, to ensure the integrity and confidentiality of applications and data in use and at rest. We protect containers in special-build lightweight VMs, called Secure Container VMs (SC-VMS). TCX preserves the agility and manageability of containers by offering secure services for standard Docker containers. Using a single trusted VM per host system, TCX provides advanced security services to all SC-VMS, including secure deployment, secure remote access, secure storage and secure communication between SC-VMS. Contributions. The main contributions of our work are: • We present TCX, a novel security architecture for secure containers in the cloud. TCX provides integrity and confidentiality guarantees for containers executed in untrusted clouds at all times. • Our implementation of the TCX architecture provides seamless integration into Docker, based on AMD SEV and the Kata Containers project. • TCX provides a secure and transparent communication channel for secure containers, i.e., Docker cannot distinguish between locally or remotely executed containers. • We thoroughly evaluate our implementation with respect to security and performance aspects. We analyze relevant attack vectors and explain how TCX protects against these threats. In our performance evaluation, which shows the practicability of our implementation, we evaluate computational-intensive workloads (SPEC2017 benchmark suite), network-intensive workloads (NGINX and Apache webserver) and memory-intensive workloads (Redis inmemory database).

II. BACKGROUND
In this section, we introduce the key technologies TCX is based on, which are: Trusted Execution Environments (TEEs), the security architecture AMD Secure Encrypted Virtualization (SEV) and Kata Containers.

A. Trusted Execution Environments
Trusted Execution Environments (TEEs) [5], [8], [15], [18], [28], [30], [34] are a type of security architecture which became prominent in recent years. TEEs securely isolate workloads from their underlying host system, such that they cannot be manipulated during run time. TEE implementations provide one or more secure execution environments, often called enclaves. Enclaves run in parallel to the commodity operating system and applications, which are referred to as the Rich Execution Environment (REE). The Trusted Computing Base (TCB) includes all software and hardware which enforces the security guarantees of the TEE and which needs to be trusted inherently. TEEs try to achieve the following goals: • Strong Run-time Protection. Isolate enclaves from the REE at rest and at run time. Execution primitives, e.g., the register state or the memory of the TEE, must be isolated from the REE at all points in time. • Verifiable State. The TEE's boot process and state during must be externally verifiable at run time using attestation. • Small TCB. The TCB must be kept minimal to reduce the risk of vulnerabilities that can lead to a TEE compromise. • Backwards Compatibility. A TEE should integrate into the existing hardware and software ecosystem as easily as possible in order to promote its adoption in practice. • Low Performance Overhead. Another important factor for the acceptance of a TEE in practice is that it only introduces a low performance overhead.

B. AMD Secure Encrypted Virtualization
AMD Secure Encrypted Virtualization (SEV) [2], [3], [30] is a TEE architecture which targets cloud servers. SEV allows to protect multiple Virtual Machines (VMs) from a malicious host system or underlying hypervisor. SEV was built by reusing and extending existing features of AMD systems, namely, the Secure Virtual Machine (SVM) extension and the Secure Memory Encryption (SME) [30] extension. Moreover, a co-processor is added to the System-on-Chip (SoC), called the Platform Security Processor (PSP)In the following, we describe how these technologies are combined in SEV to protect sensitive VMs, even in the advent of strong adversaries or a malicious cloud provider.
Memory Encryption. Secure Memory Encryption (SME) [30] encrypts the complete system memory in order to prevent live RAM introspection or cold-boot attacks [27]. In SEV, the SME feature is extended to provide memory encryption for sensitive VMs, whereby the memory of each VM is encrypted with a different key in order to achieve an isolation between the VMs and also the REE. It is the VM's responsibility to define which pages are encrypted and which are shared with the hypervisor. The PSP exposes various commands to the hypervisor in order to set up SEV for a VM and to encrypt its initial boot code. The PSP generates and assigns each VM a unique VM Encryption Key (VEK), which is not accessible by any software running on the main CPU, and configures the memory controller accordingly. The binding between VM and VEK is done via the ASID which is used to index into the list of all currently usable VM encryption keys. An AES engine resides inside the memory controller, which encrypts data with 128 bit keys in Electronic Codebook (ECB) mode.
Attestation. The attestation functionality for SEV VMs is provided by the PSP which is the first component to boot on the SoC. Every AMD SoC exports various public certificates, including a public Diffie-Hellman share (PDH) and a Chip Endorsement Key (CEK) for unique identification. Before launching a VM, the VM owner can request a signed attestation report of the PSP for the initial encrypted bootcode.
If the verification of the attestation is successful, the owner can use the PDH to create a shared secret symmetric transportation key with the PSP, which is used to inject an encrypted secret into the VM. By leveraging the Diffie-Hellman scheme, only the target PSP is able to decrypt the secret and to inject the secret into the VM by encrypting it with the VEK.

C. Kata Containers
Kata Containers [17] is a project which aims to add another level of isolation to Docker containers by executing containers within VMs. The project considers an adversary who is able to break out of the software isolation environment offered by containerization, and has the capability to escalates his privileges. The underlying hypervisor, in contrast to SEV, is assumed to be trusted. By encapsulating containers in VMs, the adversary only gains access to the information of the container VM. The host system and all other container VMs remain protected. Kata Containers integrate seamlessly into the existing Docker ecosystem. Kata consists of the kata-runtime, Kata VMs, and the kata-agent.
The kata-runtime process runs on the underlying host system and receives Open Catalog Interface (OCI) compatible commands from containers (a standardized high-level container runtime), which are translated into internal commands. The kata-runtime is also responsible for setting up and starting the hypervisor and all Kata VMs, including their devices.
The Kata VM is the VM in which a container is executed. It is based on a minimal Linux image, which offers as few services as possible. Also, the kernel is compiled to only support absolutely necessary drivers, most of them virtio drivers, which represent virtualization aware devices for better performance. Kata can be configured to use one of multiple hypervisor technologies, e.g. KVM [31].
Finally, the kata-agent process runs within the Kata VM and receives commands from the kata-runtime. The kata-agent is based on runc, the default container runtime used by Docker. As a container runtime, it is responsible for isolating the container process and setting up all needed mechanisms, such as, namespaces, chroot, or seccomp.

III. ADVERSARY MODEL
We assume TCX to be implemented on off-the-shelf cloudtargeted TEEs which provide strong protection for VM workloads. Currently, the only product-ready TEE which focuses on cloud servers is AMD SEV and thus, we inherit our adversary model from its newest version, SEV-SNP. 3 SEV assumes a strong adversary, which is able to fully compromise the system software including the kernel of the host system and even the hypervisor, thus, the adversary can read and even manipulate the complete system memory during runtime. Moreover, he is able to spawn malicious host processes, commodity VMs and even SEV VMs. Also, the adversary can monitor the complete network traffic, i.e., inject packets and impersonate the host to observe the traffic send by the SEV VMs. The adversary's goal is to infiltrate the SEV VMs or to manipulate the network traffic sent to SEV VMs in order to manipulate them or extract their sensitive data. Furthermore, the adversary has physical access to the server, which allows him to perform non-invasive physical attacks such as bus snooping or cold-boot attacks [27].
Aligned with the threat model of SEV and other industry TEEs, e.g., Intel SGX [28] or Arm TrustZone [5], sidechannel attacks are considered out of scope, including microarchitectural side-channel attacks (e.g., performed on the TLB [26] or cache [13], [42], [55]), controlled side-channel attacks [35], [53], [54], and physical side-channel attacks [22], [36]. 4 Further, we consider Denial-of-Service attacks out of scope, since an adversary with full control over a system can, for example, shut down the complete system. The PSP, the memory controller and its integrated AES engine represent the hardware TCB of TCX, whereas our newly introduced Root VM, which we describe in more detail in Section V, represents TCX's software TCB. Aligned with other TEEs [5], [8], [15], [18], [28], [30], [34], we assume this small set of hardware and software components to be functioning correctly and to be inherently trusted.

IV. REQUIREMENTS ANALYSIS
In this section, we list all requirements a practical security architecture needs to fulfill, in order to providing strongly protected software containers. In the remainder of the paper, we will show how TCX meets these requirements. R1 Container Confidentiality: Confidentiality of all data & code inside containers must be ensured at all times. R2 Container Integrity: Integrity of containers must be preserved at run time and at rest. Any tampering attempts should either be detected or prevented. R3 Protection against Malicious Host and VM: Confidentiality and integrity must be provided in presence of a malicious host system as well as in presence of malicious commodity and SEV VMs (Section III). R4 Secure Communication: Secure communication channels between a container and its provider must be provided to enable a secure container management. R5 Secure Deployment and Attestation: All containers must be started in an expected state, and the container state must be remotely verifiable through attestation. R6 Flexibility and Usability: The solution must be highly usable and integrate into the existing software ecosystem in order be easily adoptable in practice. R7 Off-the-shelf Hardware: The solution should not require hardware modifications. Instead, only off-the-shelf hardware platforms should be used to also being able to upgrade already manufactured cloud servers. R8 Low Performance Overhead: The solution should induce only a moderate performance impact to achieve a reasonable trade-off between security and performance.

V. DESIGN
In this section, we describe the design of Trusted Container Extensions (TCX). We first give an overview of the system components in our architecture. Next, we describe the lifecycle of a container secured with TCX and the provided securechannel service.
The goal of TCX, as depicted in Figure 2, is to securely deploy sensitive containers on untrusted machines in cloud environments which provide TEEs [3], [6], [29]. The key idea of TCX is to execute every sensitive container in a single protected VM provided by the underlying TEE. The secure container environments, which we call Secure Container VM (SC-VM), are protected from the host system, the underlying hypervisor and all other SC-VMS. Every SC-VM is securely deployed by a trusted VM called the Root VM, which is deployed and managed by a trusted third party. In the following, we describe each system component in our architecture: • The Host System is the system which hosts the commodity (unprotected) VMs, Secure Container VMs and the Root VM. As it is part of a cloud service infrastructure, multiple Host System instances can exist. In our architecture, as described in Section III, we assume that the Host System is untrusted, as it may act malicious intentionally or could be compromised by an attacker. • The Secure Container VMs host all sensitive containers.
They utilize the protection capabilities of the underlying TEE to achieve a strong isolation from the regular Rich Execution Environment (REE) of the Host System. In TCX, the Host System is capable of hosting many SC-VMS simultaneously. • The Root VM securely deploys SC-VMS on the system, verifies their boot process and offers additional security services, such as the establishment of secure communication channels between secured containers. On each Host System, one instance of the Root VM exists. The Root VM is the trust anchor of SC-VMS, similar to the Quoting Enclave in Intel SGX. • The Container Owner is the cloud tenant who wants to deploy its sensitives container to the Host System. After deployment, TCX enables the Container Owner to securely manage its containers and exchange sensitive data with it. • The Deploy System (operated by the trusted third party) acts as a root-of-trust, and is responsible for securely deploying the Root VM to every Host System. • The TEE Primitives are an abstract representation of the combined hardware and software components which enable the TEE functionality on the platform, and which provide basic security functionalities, such as, creating attestation reports or injecting secrets into its enclaves.

A. Secure Container VM Lifecycle
The lifecycle of an SC-VM consists of 4 steps ( Figure 2). Root VM Deployment. The first step, which is only done once for every Host System, is the secure deployment of the Root VM performed by the trusted third-party Deploy System 1 . First, the Deploy Service requests an attestation report from the TEE Primitives of the Host System. After the attestation report has been validated, the Deploy Service creates an identity Cert RootV M in the form of a certificate and injects it in the Root VM via the TEE Primitives.
Secure Container VM Creation. Before an SC-VM can be created, the Container Owner uploads a confidentialityand integrity-protected container image to the Host System. In order to create an SC-VM, the Container Owner requests the Root VM to attest the newly created protected VM. Moreover, the Container Owner sends his certificate Cert Owner to the Root VM 2 . As the Container Owner can also request the certificate Cert RootV M of the Root VM, she can create an encrypted and authenticated channel for a secure communication with the container. If the attestation of the newly created SC-VM was successful, the Root VM creates a certificate Cert V M N . This certificate acts as an identity for the SC-VM. Then, the Root VM injects the secrets Cert RootV M , Cert V M N and Cert Owner into the SC-VM 3 . Hence, the Container Owner knows exactly which SC-VM was created for her and the SC-VM knows who its owner is.
Secure Container VM Communication. When the SC-VM has been booted, the Container Owner establishes a secure channel to the SC-VM 4 which is mutually authenticated using the previously distributed certificates. The channel is used by the Container Owner to securely exchange commands and data with the SC-VM.
Secure Container VM Execution. After establishing a secure communication channel to the SC-VM, the Container Owner instructs the Host System to load the protected container image into the SC-VM. Using the secure channel, the Container Owner sends the image key Key Image to the VM, which is used to load the image. After the image has been loaded, the Container Owner instructs the SC-VM to finally execute the container image.

B. Secure Channel Service
Besides secure deployment, the Root VM offers run-time services for secure-channel establishment between containers. Channel establishment uses container-level mutual authentication. It is also possible to determine the owner of the containers respective SC-VM, such that a container can also decide to only handle a subset of requests from certain owners.
In order to establish a secure communication between the two containers C 1 and C 2 , the following steps are needed.

VI. IMPLEMENTATION
We implemented a prototype of TCX for Linux using AMD Secure Encrypted Virtualization (SEV), hence, fulfilling R7. An overview of our prototype in shown Figure 1. We implemented container management using Kata Containers [17] in combination with Docker [19] as the container ecosystem. In the following, we first provide details on TCX's main system components and then, describe the SC-VM startup, the container images and TCX's role management.
Container Owner. The Container Owner system represents the cloud tenant that uses the provided cloud container service. We extended the existing Kata Containers architecture by executing the Kata VM on another system than the Container Owner's system. Running the kata-runtime on the Container Owner's system results in a completely transparent solution for Docker, such that Docker cannot differentiate between a locally running container and a remote container in the cloud. Docker only interacts with the kata-runtime via standardized Open Container Interface (OCI) commands and does not manage the execution of containers itself, but delegates this functionality to the runtime. This allows the Container Owner to manage the container state with existing tools, such that R6 is fulfilled. Normally, kata-runtime manages the hypervisor on which the SC-VM is running, as well as the kata-agent within the VM. Therefore, it is security-critical to verify authenticity of all requests sent to them and to secure the communication between them, otherwise these requests could be forged and the container could be compromised. We implement a secure proxy on both sides, such that all commands from the Container Owner are sent via an authenticated secure channel.
Host System. The Host System runs the Host Service, which manages the lifecycle of SC-VMS and configures KVM as the hypervisor accordingly. Furthermore, the Host Service includes a switch, which routes incoming connections to a destination SC-VM. Container Ownerscan reach SC-VMS and the Root VM via this exposed network interface. The Host Service is also responsible for container image management. It exposes a HTTPS server over which Container Owners can upload encrypted images. During SC-VM creation, the Host Service loads an image into a 9P filesystem, which is shared between host and VM.
Deploy System. The Deploy System hosts a Deploy Service, which builds upon sev-tool [4].
Root VM. The Root VM is responsible for SC-VM attestation. Like the Deploy Service, it uses sev-tool for this process. The Root VM provides a secure-channel-creation service for establishing secure connections between containers. Our implementation is connection-oriented, analogous to existing network programming interfaces. We provide a simple Golang API for developers. The Root VM is conceptually similar to, e.g., Intel SGX's Quoting Enclave, and can be deployed at manufacturing time via board support packages (BSPs).

A. SC-VM Startup
We implemented a Secure Boot process for every SC-VM, ensuring that every SC-VM is initially in a secure state. TCX uses PSP as the root for the secure boot chain-of-trust, which attests the UEFI boot code of each SC-VM. All attestation reports are verified by sev-tool, an official implementation for SEV attestation by AMD. We implemented UEFI using Open Virtual Machine Firmware (OVMF).After OVMF has been booted, OVMF loads the Linux Kernel into encrypted memory and calculates its SHA256 hash. We modified OVMF, such that this hash is directly embedded in OVMF, and therefore, also verified by SEV attestation. Furthermore, we modified OVMF such that it can only load a kernel with an embedded hash. We also embed the kernel parameters within the image, as passing these from the host can lead to a potential compromise of the kernel. As the kernel will later pass execution to binaries of the SC-VM filesystem, we also have to protect the SC-VM filesystem from manipulation. We leverage dm-verity for securing SC-VM images.dm-verity is the de-facto standard for filesystem integrity checks on Linux, and is, e.g., used on Android systems [24]. For ensuring integrity of the SC-VM's container images, we leverage dm-integrity in combination with dm-crypt, in order to form an authenticated encryption mechanism. We use AES with XTS as chaining mode for encryption, and HMAC-SHA256 for integrity tag computation.

B. Container Images
Docker splits container images into multiple layers, where each layer corresponds to a step in the image building process. Using this, Docker can store layers which are used by multiple images only once in order to save space. Images will then be reconstructed on-demand before the image is executed. Furthermore, Docker will add a writable top layer, which is then associated with a container instance. Since this is a key feature of Docker, our architecture also needs to implement this functionality. For this, we build upon the devicemapper functionality and its snapshot feature. Using this, we create a block overlay file in which all modifications of the encrypted container image will be stored in. Both files get loaded into the SC-VM upon start.

C. Role Management
In TCX, roles are represented by a Certification Authority hierarchy. The Root CA creates three Intermediate CAs, one for each type of entity, which sign TLS certificates for the according system types. The Intermediate Root VM CA also creates another CA, which is bound to a Root VM instance. This Root VM CA is responsible for creating SC-VM certificates, which are injected into the SC-VM and also sent to the Owner. Using this hierarchy, all systems can authenticate each other and identify the role of the opposite party. Systems can use mutual authentication where appropriate.

VII. EVALUATION
In this section, we evaluate TCX with respect to security (Section VII-A) as well as performance (Section VII-B).

A. Security Analysis
TCX provides bi-directional isolation: (1) The containers are protected from accesses by external entities, including privileged entities. (2) At the same time the host system, including the host's privileged software and other containers, are protected from unauthorized access by the container owner. Both scenarios are analyzed subsequently by distinguishing two types of adversaries.
The general concept and design of TCX can be instantiated with different TEE architectures, e.g., Intel TDX or Arm CCA. In this section we focus on the TCX implementation based on AMD SEV (Section VI). However, the security arguments apply in similar ways for other TEE architectures.
Adversary Types. Adversary Type 1 (A1) controls the host's system -either a malicious cloud provider or an attacker that gained control over parts of the host system infrastructure -and aims to gain access to a protected container. Adversary Type 2 (A2) is a malicious container owner that aims to break out of the virtualization-based sandbox.
Container Isolation. TCX leverages SEV to isolate containers. A1 would need to break the isolation guarantees of SEV, i.e., TCX's security is reduced to the security of the underlaying TEE architecture. Concretely, every SC-VM will be booted into a secure state. Furthermore, the memory controller ensures that every SEV VM uses memory which is encrypted through a different VEK. The VEK itself is never accessible to the hypervisor or other software, including other SEV VMs. OVMF and Linux are both SEV-aware, and they correctly set up all page tables in order to protect code and data from being shared with the hypervisor. The correct setup of an SC-VM is verified via attestation. Every SEV VM will be attested by another trusted entity. The Root VM is attested by the Deploy Service and all SC-VMS on a Host System are attested by the Root VM, which fulfills R5.
TCB Size. The TCB of TCX consists of the PSP and the Deploy System which is hosted by a trusted entity off platform. The Root VM which ensure the secure boot of all SC-VMS is attested and securely booted by the Deploy System. Secure Communication. All network communication is secured via TLS. This ensures communication which is integrityprotected and confidentiality-protected from possible attacks of A1. Where required, we use mutual authentication. The Container Owner and his SC-VM use mutual authentication to be ensured that they communication with the real party.
Role Impersonation. TCX encodes the roles of various system with a CA hierarchy. TCX checks that Root VM has a valid certificate which was signed by the Root VM Intermediate CA. This way, a malicious Host System A1 with a valid certificate in our CA cannot impersonate a Root VM. The certificate of the Deploy Service is validated in the same way. These mechanisms in combination with all communication secured fulfillR4.
Secure Storage. As the container images need to be uploaded to the Host System prior to execution, it is necessary to store them securely, as they might contain sensitive information. For this, we leverage cryptsetup and integritysetup in order to create an authenticated container disk file. Before the SC-VM executes the first executable of the container image, the encryption key is passed to the SC-VM. As a result, only the SC-VM and the Container Owner can decrypt the container disk file, fulfilling R1 and R2.
Fake SEV. An attack might want to fake the existence of SEV. An attacker A1 might emulate an SEV-enabled CPU, such that the SEV VMs assume to run within a genuine SEV VM. These kinds of attacks are prevented by SEV's design. Every AMD CPU has a unique certificate, which can be validated using AMD's servers. The encrypted SEV secret can only be decrypted by the PSP for which it was encrypted, as transport keys are used which in turn are encrypted with a  secret key derived from the public DH share of the PSP. Only the target PSP knows the private part of the DH share and can successfully generate the secret to decrypt the transport keys. Malicious Container Owner. Our implementation builds upon Kata Containers, which aims to isolate containers from the host system in case of a containerization escape through kernel exploits. Breaking out of the container software isolation environment does not offer a significant advantage to the adversary A2, as the SC-VM itself does not contain confidential information. The host memory, except shared MMIO, is not accessible to the SEV VM. Even if A2 is able to break out of the virtualization isolation (becoming A1), other SC-VMS cannot be infiltrated, as we previously described. Hence, R3 is fulfilled.

B. Performance Evaluation
We evaluated the performance of TCX on an SEV-capable system. We conducted experiments for computational performance, network throughput and database transaction performance, and show that our implementation satisfies R8. As we improve over Kata containers, i.e., containers running within VMs for additional security, we compare against containers running in regular VMs without SEV.
Experiment Setup. For our experiments, we used a DELL PowerEdge R615 server with an AMD EPYC 7262 8-core CPU, 32 GB memory and a 512 GB SATA SSD. We connected the server via a 1 GiB Ethernet to another system, which we ran the network throughput benchmarking tools, in order to eliminate any bottlenecks from running benchmarking tools on the same system. For this system, we used a Lenovo ThinkStation P330, equipped with an Intel i7-8700K 6-core CPU, 16GB memory and a 128 GB SATA SSD.
NGINX. As containers are often used to deploy web services, we first evaluate the currently most popular and widely distributed webserver NGINX. NGINX follows an asynchronous event-driven approach for connection handling and was explicitly developed for high performance. For throughput benchmarking, we use the Siege benchmark tool [46]. For a more comprehensive view on how SC-VMS perform, we evaluate these benchmarks with different amounts of resources  assigned to the VMs. Table I and Figure 3 show the resourceassignment and the results of our evaluation for NGINX. A standard VM induces an overhead of 8.82% on NGINX request throughput, compared to the results of an unmodified container running NGINX on the host. A SC-VM induces an overhead of 22.1% on throughput, which results in 11.82% overhead relative to the standard VM. As NGINX frequently accesses memory, SEV's memory encryption is a likely cause for this overhead, especially due to the high number of requests/s that NGINX can manage. Apache. We also evaluate the performance impact on Apache, another popular webserver. Apache forks its webserver process upon an incoming request, which leads to higher memory consumption than in event-driven model favored by NGINX. The evaluation results are shown in Table II. VMs induce an overhead of 7.21% for Apache, while SC-VMS induce an overhead of 13.36%. Compared to the normal VM, the SC-VMS induces an average overhead of 7.77%. Interestingly, we observe performance gains for SC-VMS with specific resource configuration in Figure 3, which we attribute to special beneficial scheduling and memory accesses conditions occurring when encapsulating Apache in a VM, outweighing the costs of memory encryption.
Redis. Redis is an in-memory key-value database service. In order to persistently store data, Redis forks itself and writes the state to storage in the background. The service supports a large variety of data structures, such as lists, hashtables and sets. Redis offers a benchmarking tool, called redis-benchmark, which we used for benchmarking Redis within a SC-VM. Figure 4 shows the performance impact when running Redis within a normal non-SEV VM an within a SC-VM, both assigned with 4 vCPUs and 8 GB RAM. During experiments, it showed that Redis does not benefit from increased vCPUs and memory, in contrary to NGINX and Apache. Redis is highly dependent on single core performance. In our evaluation, we use 255 concurrent connections and a default data size of 256 bytes. As our results show, a non-SEV VM introduces an overhead of 11.95%, while a SC-VM introduces a overhead of 12.09%.
Use-Case: NGINX with CGI and MariaDB Database.  Secure Channel. We also evaluated TCX's secure channel service for secure inter-container communication separately. For this, we developed a custom benchmarking tool, which repeatedly creates secure channel connection and sends a HTTP request. On average, a SC-VM was able to create 72.97 new connections per second. Further testing indicated that the bottleneck is during the data sending phase rather than the connection creation or TLS handshake phase. We attribute this behavior to our userland implementation. Userland implementations tend to have a drastic performance impact in contrast to virtualization-aware solutions. For example, we also evaluated network throughput for SC-VMS with nonvirtualization-aware devices our experiments showed that this setup only achieved around 15% of the performance of the virtualization-aware setup. We believe that such a speedup is also possible for the secure channel.
SPEC2017. We further evaluate TCX by measuring the computational performance impact introduced by SEV. For measuring computational overhead, we use SPEC2017, a commonly used benchmark tool. In order to thoroughly evaluate the overhead added by each isolation stage, we evaluate the overhead of virtualization, SEV and containerization within a SEV VM. We measured an average overhead of 4.92% for a non-SEV VM. A SEV VM introduces an average overhead of 5.19%.The SC-VM introduces a slightly higher average overhead of 5.77%. Hence, TCX's additional software isolation layer only introduces a very minor performance overhead.

C. Memory and Storage Overhead
In Table III, we compare the disk usage and memory consumption during idle and load with a normal Ubuntu 20.04 VM. We further compare these results to the resource usage of SC-VMS. SC-VMS have a higher memory usage than a normal VM, since the usage of the devicemapper functionalities of the kernel requires additional memory. Our experiments showed that a SC-VM needs 1GB RAM in order to boot and to be able to load an encrypted container image.

VIII. RELATED WORK
In this section, we compare TCX to related work, which we divide into the general approaches to protect legacy code using TEEs, and those works focusing on protecting containers using TEEs. Trusted Execution Environments (TEEs), in general, have been subject to extensive research in industry [5], [28], [30], as well as in the academic community [8], [15], [18], [34], [39] and are therefore not discussed in this section.

A. Protecting Legacy Code
A significant hurdle for adoption of confidential computing is the requirement to adapt code. While various examples of successful ports of complex services to TEEs exist [10], [11], [23], [51], it requires expert knowledge about TEEs. Hence, an ongoing trend is to enable unmodified software to run in a TEE. However, TEEs either only offer user-space execution, or require the deployment of an additional kernel and respective drivers by the developer.
Graphene-SGX [50], SGX-LKL [43], and Occlum [44] aim to securely offer kernel services to enclaves, without the need to completely emulate or execute another kernel. They leverage a so-called Library OS, which acts as a shielding layer for syscall services. Besides better performance, these approaches also offer a smaller TCB. For instance, Graphene-SGX implements most of kernel functionality in enclave code and syscalls are passed through to the host OS as needed.
In order to further secure SGX enclaves, PANOPLY [45] aims to reduce the TCB by splitting enclaves into multiple micro containers, called microns. PANOPLY also ensures secure inter-micron control-flow and data-flow and also provides a shielding layer for secure syscalls.
Nonetheless, all solutions building on Intel SGX suffer from the same problems. For one, Intel SGX requires code to be signed by Intel to load in release mode, preventing execution of user-supplied code. In order to circumvent this restriction, all above listed solutions mark a code region as writable and executable, and load the user-supplied code into this region. However, a restriction of SGX is that memory pages can set their attributes only once. This leaves all loaded usercode on pages with full permissions and introduces code injection attacks, which have been mitigated in regular software for over a decade. In addition, as all evaluations show, SGX-based solutions suffer from a significant performance impact.
Instead of relying on SGX, SEVGuard [41] isolates userspace applications by leveraging SEV. As the previously introduces approaches, SEVGuard also passes the syscalls to the host OS. However, SEVGuard does not suffer from the memory attribution problem, since SEV offers the capability to change memory attributes at any point in time. SEVGuard does not implement a secure storage mechanism or a shielding layer, which makes it vulnerable to Iago attacks. Another vendor-independent virtualization approach to isolate applications is Sego [33], which runs the OS and the program in different VMs. In order to handle syscalls within the user enclave, Sego also introduces a shielding layer. In contrast, TCX is able to directly handle system calls within the enclave.

B. TEE Container Runtimes
SCONE [7] and TZ-Container [56] focus on securing containers from untrusted host systems. SCONE offers, like Graphene-SGX, a Library OS, which implements syscall functionalities and a shielding layer for host services. SCONE provides the integration of Docker containers in secure SGX enclaves. However, as other solutions, SCONE suffers from various problem introduced by Intel SGX, in particular, SGX enclaves have inflexible memory management (allocated at system boot) and developers need to explicitly implement SGX-specific multithreading by spawning threads in the host process that all enter the enclave. Further, due to SCONE's user-space threading model, SCONE cannot handle certain system calls such as exec or fork, hence, requiring modifications in many legacy applications. In contrast, TCX leverages its own kernel in the Secure Container VM, handles system calls directly in the VM, and hence, offers near-native system call handling speeds. Also, TCX provides regular threading that does not require changes to the SC-VMS, and provides highly-flexible resource allocation.
TZ-Container leverages management components in the TrustZone secure world to provide stronger isolation for containers and prevent page table modifications after configuration. For that, the normal-world kernel needs to be modified such that all page table modifications are trapped to the secureworld component. However, this directly implies that TZ-Container cannot protect from an untrusted cloud provider, as the kernel can be replaced with a non-trapping version. Contrary to TZ-Container, TCX also protects against malicious cloud providers and does not require kernel modifications.

IX. CONCLUSION
In this paper, we presented TCX, an architecture for secure deployment of containers on untrusted cloud systems. We leverage AMD SEV for our implementation, however, as our design is generic, it is also possible to support other TEE solutions if they meet the same requirements we outlined, e.g., the recently announced Armv9-A Confidential Computing Architecture (CCA), or Intel's recently announced Trust Domain Extensions (TDX). Both of these architectures are the counterparts to SEV for Arm and Intel CPUs. Since they support the features needed for our architecture, it is possible to implement our architecture also on those security architectures. Finally, we showed that our architecture protects against many attack scenarios. We guarantee secure deployment of containers, their protection at runtime and to keep all data within the containers safe at all times. As our solutions builds on Kata Containers, it integrates seamlessly into the Docker architecture. Our evaluation shows a performance impact of 5.77%, and a network throughput overhead of 22.1% for NGINX and an overhead of 13.36% for Apache.