 steps while using on-demand instances for others.If no step-specific settings are specified, the orchestrator will use the resources specified in the orchestrator settings for each step and run the entire pipeline in one VM. If step-specific settings are specified, an orchestrator VM will be spun up first, which will subsequently spin out new VMs dependant on the step settings. You can disable this behavior by setting the disable_step_based_settings parameter to True in the orchestrator configuration, using the following command:

zenml orchestrator update <ORCHESTRATOR_NAME> --disable_step_based_settings=True

Here's an example of how to configure specific resources for a step for the AWS cloud:

from zenml.integrations.skypilot_aws.flavors.skypilot_orchestrator_aws_vm_flavor import SkypilotAWSOrchestratorSettings

# Settings for a specific step that requires more resources

high_resource_settings = SkypilotAWSOrchestratorSettings(

instance_type='t2.2xlarge',

cpus=8,

memory=32,

use_spot=False,

region='us-east-1',

# ... other settings

@step(settings={"orchestrator.vm_aws": high_resource_settings})

def my_resource_intensive_step():

# Step implementation

pass

When configuring pipeline or step-specific resources, you can use the settings parameter to specifically target the orchestrator flavor you want to use orchestrator.STACK_COMPONENT_FLAVOR and not orchestrator component name orchestrator.STACK_COMPONENT_NAME. For example, if you want to configure resources for the vm_gcp flavor, you can use settings={"orchestrator.vm_gcp": ...}.

By using the settings parameter, you can tailor the resources for each step according to its specific needs. This flexibility allows you to optimize your pipeline execution for both performance and cost.

Check out the SDK docs for a full list of available attributes and this docs page for more information on how to specify settings.

PreviousAirflow Orchestrator

NextHyperAI Orchestrator

Last updated 15 days ago