Amazon SageMaker

Executing individual steps in SageMaker.

SageMaker offers specialized compute instances to run your training jobs and has a comprehensive UI to track and manage your models and logs. ZenML's SageMaker step operator allows you to submit individual steps to be run on Sagemaker compute instances.

When to use it

You should use the SageMaker step operator if:

one or more steps of your pipeline require computing resources (CPU, GPU, memory) that are not provided by your orchestrator.

you have access to SageMaker. If you're using a different cloud provider, take a look at the Vertex or AzureML step operators.

How to deploy it

Create a role in the IAM console that you want the jobs running in SageMaker to assume. This role should at least have the AmazonS3FullAccess and AmazonSageMakerFullAccess policies applied. Check here for a guide on how to set up this role.

Infrastructure Deployment

A Sagemaker step operator can be deployed directly from the ZenML CLI:

zenml orchestrator deploy sagemaker_step_operator --flavor=sagemaker --provider=aws ...

You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the dedicated documentation section.

How to use it

To use the SageMaker step operator, we need:

The ZenML aws integration installed. If you haven't done so, runCopyzenml integration install aws

Docker installed and running.

An IAM role with the correct permissions. See the deployment section for detailed instructions.

An AWS container registry as part of our stack. Take a look here for a guide on how to set that up.