ster <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --setLambda Labs is a cloud provider that offers GPU instances for machine learning workloads. Unlike the major cloud providers, with Lambda Labs we don't need to configure a service connector to authenticate with the cloud provider. Instead, we can directly use API keys to authenticate with the Lambda Labs API.

zenml integration install skypilot_lambda

Once the integration is installed, we can register the orchestrator with the following command:

# For more secure and recommended way, we will register the API key as a secret

zenml secret create lambda_api_key --scope user --api_key=<VALUE_1>

# Register the orchestrator

zenml orchestrator register <ORCHESTRATOR_NAME> --flavor vm_lambda --api_key={{lambda_api_key.api_key}}

# Register and activate a stack with the new orchestrator

zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set

step-specific settings.

While testing the orchestrator, we noticed that the Lambda Labs orchestrator does not support the down flag. This means the orchestrator will not automatically tear down the cluster after all jobs finish. We recommend manually tearing down the cluster after all jobs finish to avoid unnecessary costs.

Additional Configuration

For additional configuration of the Skypilot orchestrator, you can pass Settings depending on which cloud you are using which allows you to configure (among others) the following attributes:

instance_type: The instance type to use.

cpus: The number of CPUs required for the task. If a string, must be a string of the form '2' or '2+', where the + indicates that the task requires at least 2 CPUs.

memory: The amount of memory in GiB required. If a string, must be a string of the form '16' or '16+', where the + indicates that the task requires at least 16 GB of memory.