How to configure a pipeline with a YAML

Specify a configuration file

All configuration that can be specified in a YAML file can also be specified in code itself. However, it is best practice to use a YAML file to separate config from code.

You can use the with_options(config_path=<PATH_TO_CONFIG>) pattern to apply your configuration to a pipeline. Here is a minimal example of using a file based configuration yaml.

enable_cache: False

# Configure the pipeline parameters

parameters:

dataset_name: "best_dataset"

steps:

load_data:  # Use the step name here

enable_cache: False  # same as @step(enable_cache=False)

from zenml import step, pipeline

@step

def load_data(dataset_name: str) -> dict:

...

@pipeline  # This function combines steps together

def simple_ml_pipeline(dataset_name: str):

load_data(dataset_name)

if __name__=="__main__":

simple_ml_pipeline.with_options(config_path=<INSERT_PATH_TO_CONFIG_YAML>)()

The above would run the simple_ml_pipeline with cache disabled for load_data and the parameter dataset_name set to best_dataset.

PreviousUse configuration files

NextWhat can be configured

Last updated 15 days ago