Handle custom data types

Using materializers to pass custom data types through steps.

A ZenML pipeline is built in a data-centric way. The outputs and inputs of steps define how steps are connected and the order in which they are executed. Each step should be considered as its very own process that reads and writes its inputs and outputs from and to the artifact store. This is where materializers come into play.

A materializer dictates how a given artifact can be written to and retrieved from the artifact store and also contains all serialization and deserialization logic. Whenever you pass artifacts as outputs from one pipeline step to other steps as inputs, the corresponding materializer for the respective data type defines how this artifact is first serialized and written to the artifact store, and then deserialized and read in the next step.

Built-In Materializers

ZenML already includes built-in materializers for many common data types. These are always enabled and are used in the background without requiring any user interaction / activation:

Materializer Handled Data Types Storage Format BuiltInMaterializer bool , float , int , str , None .json BytesInMaterializer bytes .txt BuiltInContainerMaterializer dict , list , set , tuple Directory NumpyMaterializer np.ndarray .npy PandasMaterializer pd.DataFrame , pd.Series .csv (or .gzip if parquet is installed) PydanticMaterializer pydantic.BaseModel .json ServiceMaterializer zenml.services.service.BaseService .json StructuredStringMaterializer zenml.types.CSVString , zenml.types.HTMLString , zenml.types.MarkdownString .csv / .html / .md (depending on type)