Configuration hierarchy

When things can be configured on the pipeline and step level, the step configuration overrides the pipeline.

There are a few general rules when it comes to settings and configurations that are applied in multiple places. Generally the following is true:

Configurations in code override configurations made inside of the yaml file

Configurations at the step level override those made at the pipeline level

In case of attributes the dictionaries are merged

from zenml import pipeline, step

from zenml.config import ResourceSettings

@step

def load_data(parameter: int) -> dict:

...

@step(settings={"resources": ResourceSettings(gpu_count=1, memory="2GB")})

def train_model(data: dict) -> None:

...

@pipeline(settings={"resources": ResourceSettings(cpu_count=2, memory="1GB")})

def simple_ml_pipeline(parameter: int):

...

# ZenMl merges the two configurations and uses the step configuration to override

# values defined on the pipeline level

train_model.configuration.settings["resources"]

# -> cpu_count: 2, gpu_count=1, memory="2GB"

simple_ml_pipeline.configuration.settings["resources"]

# -> cpu_count: 2, memory="1GB"

PreviousRuntime settings for Docker, resources, and stack components

NextFind out which configuration was used for a run

Last updated 19 days ago