
zenml model version list breast_cancer_classifierThe ZenML Cloud ships with a Model Control Plane dashboard where you can visualize all the versions:

Passing parameters

The last part of the config YAML is the parameters key:

# Configure the pipeline

parameters:

model_type: "rf"  # Choose between rf/sgd

This parameters key aligns with the parameters that the pipeline expects. In this case, the pipeline expects a string called model_type that will inform it which type of model to use:

@pipeline

def training_pipeline(model_type: str):

...

So you can see that the YAML config is fairly easy to use and is an important part of the codebase to control the execution of our pipeline. You can read more about how to configure a pipeline in the how to section, but for now, we can move on to scaling our pipeline.

Scaling compute on the cloud

When we ran our pipeline with the above config, ZenML used some sane defaults to pick the resource requirements for that pipeline. However, in the real world, you might want to add more memory, CPU, or even a GPU depending on the pipeline at hand.

This is as easy as adding the following section to your local training_rf.yaml file:

# These are the resources for the entire pipeline, i.e., each step

settings:

...

# Adapt this to vm_azure or vm_gcp accordingly

orchestrator.vm_aws:

memory: 32 # in GB

...

steps:

model_trainer:

settings:

orchestrator.vm_aws:

cpus: 8

Here we are configuring the entire pipeline with a certain amount of memory, while for the trainer step we are additionally configuring 8 CPU cores. The orchestrator.vm_aws key corresponds to the SkypilotBaseOrchestratorSettings class in the Python SDK. You can adapt it to vm_gcp or vm_azure depending on which flavor of skypilot you have configured.

Read more about settings in ZenML here.

Now let's run the pipeline again:

python run.py --training-pipeline