Passing artifacts between pipelines

Structuring an MLOps project

An MLOps project can often be broken down into many different pipelines. For example:

A feature engineering pipeline that prepares raw data into a format ready to get trained.

A training pipeline that takes input data from a feature engineering pipeline and trains a models on it.

An inference pipeline that runs batch predictions on the trained model and often takes pre-processing from the training pipeline.

A deployment pipeline that deploys a trained model into a production endpoint.

The lines between these pipelines can often get blurry: Some use cases call for these pipelines to be merged into one big pipeline. Others go further and break the pipeline down into even smaller chunks. Ultimately, the decision of how to structure your pipelines depends on the use case and requirements of the project.

No matter how you design these pipelines, one thing stays consistent: you will often need to transfer or share information (in particular artifacts, models, and metadata) between pipelines. Here are some common patterns that you can use to help facilitate such an exchange:

Pattern 1: Artifact exchange between pipelines through Client

Let's say we have a feature engineering pipeline and a training pipeline. The feature engineering pipeline is like a factory, pumping out many different datasets. Only a few of these datasets should be selected to be sent to the training pipeline to train an actual model.

In this scenario, the ZenML Client can be used to facilitate such an exchange:

from zenml import pipeline

from zenml.client import Client

@pipeline

def feature_engineering_pipeline():

dataset = load_data()

# This returns artifacts called "iris_training_dataset" and "iris_testing_dataset"

train_data, test_data = prepare_data()

@pipeline

def training_pipeline():

client = Client()

# Fetch by name alone - uses the latest version of this artifact