pe-specific metadata and visualizations.

MetadataAll output artifacts saved through ZenML will automatically have certain datatype-specific metadata saved with them. NumPy Arrays, for instance, always have their storage size, shape, dtype, and some statistical properties saved with them. You can access such metadata via the run_metadata attribute of an output, e.g.:

output_metadata = output.run_metadata

storage_size_in_bytes = output_metadata["storage_size"].value

We will talk more about metadata in the next section.

Visualizations

ZenML automatically saves visualizations for many common data types. Using the visualize() method you can programmatically show these visualizations in Jupyter notebooks:

output.visualize()

If you're not in a Jupyter notebook, you can simply view the visualizations in the ZenML dashboard by running zenml up and clicking on the respective artifact in the pipeline run DAG instead. Check out the artifact visualization page to learn more about how to build and view artifact visualizations in ZenML!

Fetching information during run execution

While most of this document has focused on fetching objects after a pipeline run has been completed, the same logic can also be used within the context of a running pipeline.

This is often desirable in cases where a pipeline is running continuously over time and decisions have to be made according to older runs.

For example, this is how we can fetch the last pipeline run of the same pipeline from within a ZenML step:

from zenml import get_step_context

from zenml.client import Client

@step

def my_step():

# Get the name of the current pipeline run

current_run_name = get_step_context().pipeline_run.name

# Fetch the current pipeline run

current_run = Client().get_pipeline_run(current_run_name)

# Fetch the previous run of the same pipeline

previous_run = current_run.pipeline.runs[1]  # index 0 is the current run