Great Expectations

How to use Great Expectations to run data quality checks in your pipelines and document the results

The Great Expectations Data Validator flavor provided with the ZenML integration uses Great Expectations to run data profiling and data quality tests on the data circulated through your pipelines. The test results can be used to implement automated corrective actions in your pipelines. They are also automatically rendered into documentation for further visual interpretation and evaluation.

When would you want to use it?

Great Expectations is an open-source library that helps keep the quality of your data in check through data testing, documentation, and profiling, and to improve communication and observability. Great Expectations works with tabular data in a variety of formats and data sources, of which ZenML currently supports only pandas.DataFrame as part of its pipelines.

You should use the Great Expectations Data Validator when you need the following data validation features that are possible with Great Expectations:

Data Profiling: generates a set of validation rules (Expectations) automatically by inferring them from the properties of an input dataset.

Data Quality: runs a set of predefined or inferred validation rules (Expectations) against an in-memory dataset.

Data Docs: generate and maintain human-readable documentation of all your data validation rules, data quality checks and their results.

You should consider one of the other Data Validator flavors if you need a different set of data validation features.

How do you deploy it?

The Great Expectations Data Validator flavor is included in the Great Expectations ZenML integration, you need to install it on your local machine to be able to register a Great Expectations Data Validator and add it to your stack:

zenml integration install great_expectations -y