Create an ML pipeline

Start with the basics of steps and pipelines.

In the quest for production-ready ML models, workflows can quickly become complex. Decoupling and standardizing stages such as data ingestion, preprocessing, and model evaluation allows for more manageable, reusable, and scalable processes. ZenML pipelines facilitate this by enabling each stage—represented as Steps—to be modularly developed and then integrated smoothly into an end-to-end Pipeline.

Leveraging ZenML, you can create and manage robust, scalable machine learning (ML) pipelines. Whether for data preparation, model training, or deploying predictions, ZenML standardizes and streamlines the process, ensuring reproducibility and efficiency.

Start with a simple ML pipeline

Let's jump into an example that demonstrates how a simple pipeline can be set up in ZenML, featuring actual ML components to give you a better sense of its application.

from zenml import pipeline, step

@step

def load_data() -> dict:

"""Simulates loading of training data and labels."""

training_data = [[1, 2], [3, 4], [5, 6]]

labels = [0, 1, 0]

return {'features': training_data, 'labels': labels}

@step

def train_model(data: dict) -> None:

"""

A mock 'training' process that also demonstrates using the input data.

In a real-world scenario, this would be replaced with actual model fitting logic.

"""

total_features = sum(map(sum, data['features']))

total_labels = sum(data['labels'])

print(f"Trained model using {len(data['features'])} data points. "

f"Feature sum is {total_features}, label sum is {total_labels}")

@pipeline

def simple_ml_pipeline():

"""Define a pipeline that connects the steps."""

dataset = load_data()

train_model(dataset)

if __name__ == "__main__":

run = simple_ml_pipeline()

# You can now use the `run` object to see steps, outputs, etc.

@step is a decorator that converts its function into a step that can be used within a pipeline