 out the SDK Docs .

S3 data access in ZenML stepsIn Sagemaker jobs, it is possible to access data that is located in S3. Similarly, it is possible to write data from a job to a bucket. The ZenML Sagemaker orchestrator supports this via the SagemakerOrchestratorSettings and hence at component, pipeline, and step levels.

Import: S3 -> job

Importing data can be useful when large datasets are available in S3 for training, for which manual copying can be cumbersome. Sagemaker supports File (default) and Pipe mode, with which data is either fully copied before the job starts or piped on the fly. See the Sagemaker documentation referenced above for more information about these modes.

Note that data import and export can be used jointly with processor_args for maximum flexibility.

A simple example of importing data from S3 to the Sagemaker job is as follows:

sagemaker_orchestrator_settings = SagemakerOrchestratorSettings(

input_data_s3_mode="File",

input_data_s3_uri="s3://some-bucket-name/folder"

In this case, data will be available at /opt/ml/processing/input/data within the job.

It is also possible to split your input over channels. This can be useful if the dataset is already split in S3, or maybe even located in different buckets.

sagemaker_orchestrator_settings = SagemakerOrchestratorSettings(

input_data_s3_mode="File",

input_data_s3_uri={

"train": "s3://some-bucket-name/training_data",

"val": "s3://some-bucket-name/validation_data",

"test": "s3://some-other-bucket-name/testing_data"

Here, the data will be available in /opt/ml/processing/input/data/train, /opt/ml/processing/input/data/val and /opt/ml/processing/input/data/test.

In the case of using Pipe for input_data_s3_mode, a file path specifying the pipe will be available as per the description written here . An example of using this pipe file within a Python script can be found here .

Export: job -> S3