urns it with the configuration of the cloud stack.Based on the stack info and pipeline specification, the client builds and pushes an image to the container registry. The image contains the environment needed to execute the pipeline and the code of the steps.

The client creates a run in the orchestrator. For example, in the case of the Skypilot orchestrator, it creates a virtual machine in the cloud with some commands to pull and run a Docker image from the specified container registry.

The orchestrator pulls the appropriate image from the container registry as it's executing the pipeline (each step has an image).

As each pipeline runs, it stores artifacts physically in the artifact store. Of course, this artifact store needs to be some form of cloud storage.

As each pipeline runs, it reports status back to the ZenML server and optionally queries the server for metadata.

Provisioning and registering a Skypilot orchestrator alongside a container registry

While there are detailed docs on how to set up a Skypilot orchestrator and a container registry on each public cloud, we have put the most relevant details here for convenience:

In order to launch a pipeline on AWS with the SkyPilot orchestrator, the first thing that you need to do is to install the AWS and Skypilot integrations:

zenml integration install aws skypilot_aws -y

Before we start registering any components, there is another step that we have to execute. As we explained in the previous section, components such as orchestrators and container registries often require you to set up the right permissions. In ZenML, this process is simplified with the use of Service Connectors. For this example, we need to use the IAM role authentication method of our AWS service connector:

AWS_PROFILE=<AWS_PROFILE> zenml service-connector register cloud_connector --type aws --auto-configure

Once the service connector is set up, we can register a Skypilot orchestrator:

zenml orchestrator register skypilot_orchestrator -f vm_aws