board.

The Great Expectation's data profiler stepThe standard Great Expectation's data profiler step builds an Expectation Suite automatically by running a UserConfigurableProfiler on an input pandas.DataFrame dataset. The generated Expectation Suite is saved in the Great Expectations Expectation Store, but also returned as an ExpectationSuite artifact that is versioned and saved in the ZenML Artifact Store. The step automatically rebuilds the Data Docs.

At a minimum, the step configuration expects a name to be used for the Expectation Suite:

from zenml.integrations.great_expectations.steps import (

great_expectations_profiler_step,

ge_profiler_step = great_expectations_profiler_step.with_options(

parameters={

"expectation_suite_name": "steel_plates_suite",

"data_asset_name": "steel_plates_train_df",

The step can then be inserted into your pipeline where it can take in a pandas dataframe, e.g.:

from zenml import pipeline

docker_settings = DockerSettings(required_integrations=[SKLEARN, GREAT_EXPECTATIONS])

@pipeline(settings={"docker": docker_settings})

def profiling_pipeline():

"""Data profiling pipeline for Great Expectations.

The pipeline imports a reference dataset from a source then uses the builtin

Great Expectations profiler step to generate an expectation suite (i.e.

validation rules) inferred from the schema and statistical properties of the

reference dataset.

Args:

importer: reference data importer step

profiler: data profiler step

"""

dataset, _ = importer()

ge_profiler_step(dataset)

profiling_pipeline()

As can be seen from the step definition , the step takes in a pandas.DataFrame dataset, and it returns a Great Expectations ExpectationSuite object:

@step

def great_expectations_profiler_step(

dataset: pd.DataFrame,

expectation_suite_name: str,

data_asset_name: Optional[str] = None,

profiler_kwargs: Optional[Dict[str, Any]] = None,

overwrite_existing_suite: bool = True,

) -> ExpectationSuite:

...