Assessing Data Quality of Annotations with Krippendorff Alpha For
  Applications in Computer Vision
  Current supervised deep learning frameworks rely on annotated data for
modeling the underlying data distribution of a given task. In particular for
computer vision algorithms powered by deep learning, the quality of annotated
data is the most critical factor in achieving the desired algorithm
performance. Data annotation is, typically, a manual process where the
annotator follows guidelines and operates in a best-guess manner. Labeling
criteria among annotators can show discrepancies in labeling results. This may
impact the algorithm inference performance. Given the popularity and widespread
use of deep learning among computer vision, more and more custom datasets are
needed to train neural networks to tackle different kinds of tasks.
Unfortunately, there is no full understanding of the factors that affect
annotated data quality, and how it translates into algorithm performance. In
this paper we studied this problem for object detection and recognition.We
conducted several data annotation experiments to measure inter annotator
agreement and consistency, as well as how the selection of ground truth impacts
the perceived algorithm performance.We propose a methodology to monitor the
quality of annotations during the labeling of images and how it can be used to
measure performance. We also show that neglecting to monitor the annotation
process can result in significant loss in algorithm precision. Through these
experiments, we observe that knowledge of the labeling process, training data,
and ground truth data used for algorithm evaluation are fundamental components
to accurately assess trustworthiness of an AI system.
