Airflow Orchestrator

Orchestrating your pipelines to run on Airflow.

ZenML pipelines can be executed natively as Airflow DAGs. This brings together the power of the Airflow orchestration with the ML-specific benefits of ZenML pipelines. Each ZenML step runs in a separate Docker container which is scheduled and started using Airflow.

If you're going to use a remote deployment of Airflow, you'll also need a remote ZenML deployment.

When to use it

You should use the Airflow orchestrator if

you're looking for a proven production-grade orchestrator.

you're already using Airflow.

you want to run your pipelines locally.

you're willing to deploy and maintain Airflow.

How to deploy it

The Airflow orchestrator can be used to run pipelines locally as well as remotely. In the local case, no additional setup is necessary.

There are many options to use a deployed Airflow server:

Use one of ZenML's Airflow stack recipes. This is the simplest solution to get ZenML working with Airflow, as the recipe also takes care of additional steps such as installing required Python dependencies in your Airflow server environment.

Use a managed deployment of Airflow such as Google Cloud Composer , Amazon MWAA, or Astronomer.

Deploy Airflow manually. Check out the official Airflow docs for more information.

If you're not using mlstacks to deploy Airflow, there are some additional Python packages that you'll need to install in the Python environment of your Airflow server:

pydantic~=2.7.1: The Airflow DAG files that ZenML creates for you require Pydantic to parse and validate configuration files.

apache-airflow-providers-docker or apache-airflow-providers-cncf-kubernetes, depending on which Airflow operator you'll be using to run your pipeline steps. Check out this section for more information on supported operators.

How to use it

To use the Airflow orchestrator, we need: