Reference secrets in stack configuration

Reference secrets in stack component attributes and settings

Some of the components in your stack require you to configure them with sensitive information like passwords or tokens, so they can connect to the underlying infrastructure. Secret references allow you to configure these components in a secure way by not specifying the value directly but instead referencing a secret by providing the secret name and key. Referencing a secret for the value of any string attribute of your stack components, simply specify the attribute using the following syntax: {{<SECRET_NAME>.<SECRET_KEY>}}

For example:

# Register a secret called `mlflow_secret` with key-value pairs for the

# username and password to authenticate with the MLflow tracking server

# Using central secrets management

zenml secret create mlflow_secret \

--username=admin \

--password=abc123

# Then reference the username and password in our experiment tracker component

zenml experiment-tracker register mlflow \

--flavor=mlflow \

--tracking_username={{mlflow_secret.username}} \

--tracking_password={{mlflow_secret.password}} \

...

When using secret references in your stack, ZenML will validate that all secrets and keys referenced in your stack components exist before running a pipeline. This helps us fail early so your pipeline doesn't fail after running for some time due to some missing secret.

This validation by default needs to fetch and read every secret to make sure that both the secret and the specified key-value pair exist. This can take quite some time and might fail if you don't have permission to read secrets.

You can use the environment variable ZENML_SECRET_VALIDATION_LEVEL to disable or control the degree to which ZenML validates your secrets:

Setting it to NONE disables any validation.