Runtime settings for Docker, resources, and stack components

Using settings to configure runtime configuration.

Part of the configuration of a pipeline are its Settings. These allow you to configure runtime configurations for stack components and pipelines. Concretely, they allow you to configure:

The resources required for a step

Configuring the containerization process of a pipeline (e.g. What requirements get installed in the Docker image)

Stack component-specific configuration, e.g., if you have an experiment tracker passing in the name of the experiment at runtime

You will learn about all of the above in more detail later, but for now, let's try to understand that all of this configuration flows through one central concept called BaseSettings. (From here on, we use settings and BaseSettings as analogous in this guide).

Types of settings

Settings are categorized into two types:

General settings that can be used on all ZenML pipelines. Examples of these are:DockerSettings to specify Docker settings.ResourceSettings to specify resource settings.

Stack-component-specific settings: These can be used to supply runtime configurations to certain stack components (key= <COMPONENT_CATEGORY>.<COMPONENT_FLAVOR>). Settings for components not in the active stack will be ignored. Examples of these are:SkypilotAWSOrchestratorSettings to specify Skypilot settings (works for SkypilotGCPOrchestratorSettings and SkypilotAzureOrchestratorSettings as well).KubeflowOrchestratorSettings to specify Kubeflow settings.MLflowExperimentTrackerSettings to specify MLflow settings.WandbExperimentTrackerSettings to specify W&B settings.WhylogsDataValidatorSettings to specify Whylogs settings.SagemakerStepOperatorSettings to specify AWS Sagemaker step operator settings.VertexStepOperatorSettings to specify GCP Vertex step operator settings.AzureMLStepOperatorSettings to specify AzureML step operator settings.