docs.

failure_hook_source and success_hook_sourceThe source of the failure and success hooks can be specified.

Step-specific configuration

A lot of pipeline-level configuration can also be applied at a step level (as we have already seen with the enable_cache flag). However, there is some configuration that is step-specific, meaning it cannot be applied at a pipeline level, but only at a step level.

experiment_tracker: Name of the experiment_tracker to enable for this step. This experiment_tracker should be defined in the active stack with the same name.

step_operator: Name of the step_operator to enable for this step. This step_operator should be defined in the active stack with the same name.

outputs: This is configuration of the output artifacts of this step. This is further keyed by output name (by default, step outputs are named output). The most interesting configuration here is the materializer_source, which is the UDF path of the materializer in code to use for this output (e.g. materializers.some_data.materializer.materializer_class). Read more about this source path here.

PreviousHow to configure a pipeline with a YAML

NextRuntime settings for Docker, resources, and stack components

Last updated 19 days ago