Kubeflow Orchestrator

Orchestrating your pipelines to run on Kubeflow.

The Kubeflow orchestrator is an orchestrator flavor provided by the ZenML kubeflow integration that uses Kubeflow Pipelines to run your pipelines.

This component is only meant to be used within the context of a remote ZenML deployment scenario. Usage with a local ZenML deployment may lead to unexpected behavior!

When to use it

You should use the Kubeflow orchestrator if:

you're looking for a proven production-grade orchestrator.

you're looking for a UI in which you can track your pipeline runs.

you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.

you're willing to deploy and maintain Kubeflow Pipelines on your cluster.

How to deploy it

The Kubeflow orchestrator supports two different modes: Local and remote. In case you want to run the orchestrator on a local Kubernetes cluster running on your machine, there is no additional infrastructure setup necessary.

If you want to run your pipelines on a remote cluster instead, you'll need to set up a Kubernetes cluster and deploy Kubeflow Pipelines:

Have an existing AWS EKS cluster set up.

Make sure you have the AWS CLI set up.

Download and install kubectl and configure it to talk to your EKS cluster using the following command:Copyaws eks --region REGION update-kubeconfig --name CLUSTER_NAME

Install Kubeflow Pipelines onto your cluster.

( optional) set up an AWS Service Connector to grant ZenML Stack Components easy and secure access to the remote EKS cluster.

Have an existing GCP GKE cluster set up.

Make sure you have the Google Cloud CLI set up first.

Download and install kubectl and configure it to talk to your GKE cluster using the following command:Copygcloud container clusters get-credentials CLUSTER_NAME

Install Kubeflow Pipelines onto your cluster.

( optional) set up a GCP Service Connector to grant ZenML Stack Components easy and secure access to the remote GKE cluster.