🪄Core concepts

Discovering the core concepts behind ZenML.

ZenML is an extensible, open-source MLOps framework for creating portable, production-ready MLOps pipelines. It's built for data scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production. In order to achieve this goal, ZenML introduces various concepts for different aspects of an ML workflow and we can categorize these concepts under three different threads:

1. DevelopmentAs a developer, how do I design my machine learning workflows?

2. ExecutionWhile executing, how do my workflows utilize the large landscape of MLOps tooling/infrastructure?

3. ManagementHow do I establish and maintain a production-grade and efficient solution?

1. Development

First, let's look at the main concepts which play a role during the development stage of an ML workflow with ZenML.

Step

Steps are functions annotated with the @step decorator. The easiest one could look like this.

@step

def step_1() -> str:

"""Returns a string."""

return "world"

These functions can also have inputs and outputs. For ZenML to work properly, these should preferably be typed.

@step(enable_cache=False)

def step_2(input_one: str, input_two: str) -> str:

"""Combines the two strings passed in."""

combined_str = f"{input_one} {input_two}"

return combined_str

Pipelines

At its core, ZenML follows a pipeline-based workflow for your projects. A pipeline consists of a series of steps, organized in any order that makes sense for your use case.

As seen in the image, a step might use the outputs from a previous step and thus must wait until the previous step is completed before starting. This is something you can keep in mind when organizing your steps.

Pipelines and steps are defined in code using Python decorators or classes. This is where the core business logic and value of your work lives, and you will spend most of your time defining these two things.