Kubernetes

Learn how to deploy ZenML pipelines on a Kubernetes cluster.

The ZenML Kubernetes Orchestrator allows you to run your ML pipelines on a Kubernetes cluster without writing Kubernetes code. It's a lightweight alternative to more complex orchestrators like Airflow or Kubeflow.

Prerequisites

To use the Kubernetes Orchestrator, you'll need:

ZenML kubernetes integration installed (zenml integration install kubernetes)

Docker installed and running

kubectl installed

A remote artifact store and container registry in your ZenML stack

A deployed Kubernetes cluster

A configured kubectl context pointing to the cluster (optional, see below)

Deploying the Orchestrator

You can deploy the orchestrator from the ZenML CLI:

zenml orchestrator deploy k8s_orchestrator --flavor=kubernetes --provider=<YOUR_PROVIDER>

Configuring the Orchestrator

There are two ways to configure the orchestrator:

Using a Service Connector to connect to the remote cluster. This is the recommended approach, especially for cloud-managed clusters. No local kubectl context is needed.

zenml orchestrator register <ORCHESTRATOR_NAME> --flavor kubernetes

zenml service-connector list-resources --resource-type kubernetes-cluster -e

zenml orchestrator connect <ORCHESTRATOR_NAME> --connector <CONNECTOR_NAME>

zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set

Configuring kubectl with a context pointing to the remote cluster and setting the kubernetes_context in the orchestrator config:

zenml orchestrator register <ORCHESTRATOR_NAME> \

--flavor=kubernetes \

--kubernetes_context=<KUBERNETES_CONTEXT>

zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set

Running a Pipeline

Once configured, you can run any ZenML pipeline using the Kubernetes Orchestrator:

python your_pipeline.py

This will create a Kubernetes pod for each step in your pipeline. You can interact with the pods using kubectl commands.