ds until the pipeline shows up in the Airflow UI).The ability to provision resources using the zenml stack up command is deprecated and will be removed in a future release. While it is still available for the Airflow orchestrator, we recommend following the steps to set up a local Airflow server manually.

Install the apache-airflow package in your Python environment where ZenML is installed.

The Airflow environment variables are used to configure the behavior of the Airflow server. The following variables are particularly important to set:

AIRFLOW_HOME: This variable defines the location where the Airflow server stores its database and configuration files. The default value is ~/airflow.

AIRFLOW__CORE__DAGS_FOLDER: This variable defines the location where the Airflow server looks for DAG files. The default value is <AIRFLOW_HOME>/dags.

AIRFLOW__CORE__LOAD_EXAMPLES: This variable controls whether the Airflow server should load the default set of example DAGs. The default value is false, which means that the example DAGs will not be loaded.

AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: This variable controls how often the Airflow scheduler checks for new or updated DAGs. By default, the scheduler will check for new DAGs every 30 seconds. This variable can be used to increase or decrease the frequency of the checks, depending on the specific needs of your pipeline.Copyexport AIRFLOW_HOME=...
export AIRFLOW__CORE__DAGS_FOLDER=...
export AIRFLOW__CORE__LOAD_EXAMPLES=false
export AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=10

# Prevent crashes during forking on MacOS
# https://github.com/apache/airflow/issues/28487
export no_proxy=*

Run airflow standalone to initialize the database, create a user, and start all components for you.

When using the Airflow orchestrator with a remote deployment, you'll additionally need:

A remote ZenML server deployed to the cloud. See the deployment guide for more information.

A deployed Airflow server. See the deployment section for more information.