ant feedback of actual contact with the raw data.)Samples generated for inference: Your model will be making predictions on real-world data being passed in. If you store and label this data, you’ll gain a valuable set of data that you can use to compare your labels with what the model was predicting, another possible way to flag drifts of various kinds. This data can then (subject to privacy/user consent) be used in retraining or fine-tuning your model.

Other ad hoc interventions: You will probably have some kind of process to identify bad labels, or to find the kinds of examples that your model finds really difficult to make correct predictions. For these, and for areas where you have clear class imbalances, you might want to do ad hoc annotation to supplement the raw materials your model has to learn from.

ZenML currently offers standard steps that help you tackle the above use cases, but the stack component and abstraction will continue to be developed to make it easier to use.

When to use it

The annotator is an optional stack component in the ZenML Stack. We designed our abstraction to fit into the larger ML use cases, particularly the training and deployment parts of the lifecycle.

The core parts of the annotation workflow include:

using labels or annotations in your training steps in a seamless way

handling the versioning of annotation data

allow for the conversion of annotation data to and from custom formats

handle annotator-specific tasks, for example, the generation of UI config files that Label Studio requires for the web annotation interface

List of available annotators

For production use cases, some more flavors can be found in specific integrations modules. In terms of annotators, ZenML features integrations with label_studio and pigeon.