 use it

To use the Airflow orchestrator, we need:The ZenML airflow integration installed. If you haven't done so, runCopyzenml integration install airflow

Docker installed and running.

The orchestrator registered and part of our active stack:

zenml orchestrator register <ORCHESTRATOR_NAME> \

--flavor=airflow \

--local=True  # set this to `False` if using a remote Airflow deployment

# Register and activate a stack with the new orchestrator

zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set

In the local case, we need to reinstall in a certain way for the local Airflow server:

pip install "apache-airflow-providers-docker<3.8.0" "apache-airflow==2.4.0" "pendulum<3.0.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.0/constraints-3.9.txt"

Please make sure to replace 3.9 with your Python (major) version in the constraints file URL given above.

Once that is installed, we can start the local Airflow server by running the following command in your terminal. See further below on an alternative way to set up the Airflow server manually since the zenml stack up command is deprecated.

zenml stack up

This command will start up an Airflow server on your local machine that's running in the same Python environment that you used to provision it. When it is finished, it will print a username and password which you can use to log in to the Airflow UI here.

As long as you didn't configure any custom value for the dag_output_dir attribute of your orchestrator, running a pipeline locally is as simple as calling:

python file_that_runs_a_zenml_pipeline.py

This call will produce a .zip file containing a representation of your ZenML pipeline to the Airflow DAGs directory. From there, the local Airflow server will load it and run your pipeline (It might take a few seconds until the pipeline shows up in the Airflow UI).