A remote container registry as part of your stack.In the remote case, the Airflow orchestrator works differently than other ZenML orchestrators. Executing a python file which runs a pipeline by calling pipeline.run() will not actually run the pipeline, but instead will create a .zip file containing an Airflow representation of your ZenML pipeline. In one additional step, you need to make sure this zip file ends up in the DAGs directory of your Airflow deployment.

ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME> which includes your code and use it to run your pipeline steps in Airflow. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.

Scheduling

You can schedule pipeline runs on Airflow similarly to other orchestrators. However, note that Airflow schedules always need to be set in the past, e.g.,:

from datetime import datetime, timedelta

from zenml.pipelines import Schedule

scheduled_pipeline = fashion_mnist_pipeline.with_options(

schedule=Schedule(

start_time=datetime.now() - timedelta(hours=1),  # start in the past

end_time=datetime.now() + timedelta(hours=1),

interval_second=timedelta(minutes=15),  # run every 15 minutes

catchup=False,

scheduled_pipeline()

Airflow UI

Airflow comes with its own UI that you can use to find further details about your pipeline runs, such as the logs of your steps. For local Airflow, you can find the Airflow UI at http://localhost:8080 by default.

If you cannot see the Airflow UI credentials in the console, you can find the password in <AIRFLOW_HOME>/standalone_admin_password.txt. AIRFLOW_HOME will usually be ~/airflow unless you've manually configured it with the AIRFLOW_HOME environment variable. You can always run airflow info to figure out the directory for the active environment.

The username will always be admin.

Additional configuration