ttings={

"orchestrator.vm_gcp": skypilot_settingsCode Example:

from zenml.integrations.skypilot_azure.flavors.skypilot_orchestrator_azure_vm_flavor import SkypilotAzureOrchestratorSettings

skypilot_settings = SkypilotAzureOrchestratorSettings(

cpus="2",

memory="16",

accelerators="V100:2",

accelerator_args={"tpu_vm": True, "runtime_version": "tpu-vm-base"},

use_spot=True,

spot_recovery="recovery_strategy",

region="West Europe",

image_id="Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest",

disk_size=100,

disk_tier="high",

cluster_name="my_cluster",

retry_until_up=True,

idle_minutes_to_autostop=60,

down=True,

stream_logs=True

@pipeline(

settings={

"orchestrator.vm_azure": skypilot_settings

Code Example:

from zenml.integrations.skypilot_lambda import SkypilotLambdaOrchestratorSettings

skypilot_settings = SkypilotLambdaOrchestratorSettings(

instance_type="gpu_1x_h100_pcie",

cluster_name="my_cluster",

retry_until_up=True,

idle_minutes_to_autostop=60,

down=True,

stream_logs=True,

docker_run_args=["--gpus=all"]

@pipeline(

settings={

"orchestrator.vm_lambda": skypilot_settings

One of the key features of the SkyPilot VM Orchestrator is the ability to run each step of a pipeline on a separate VM with its own specific settings. This allows for fine-grained control over the resources allocated to each step, ensuring that each part of your pipeline has the necessary compute power while optimizing for cost and efficiency.

Configuring Step-Specific Resources

The SkyPilot VM Orchestrator allows you to configure resources for each step individually. This means you can specify different VM types, CPU and memory requirements, and even use spot instances for certain steps while using on-demand instances for others.