Are pathologist-defined labels reproducible? Comparison of the TUPAC16
  mitotic figure dataset with an alternative set of labels
  Pathologist-defined labels are the gold standard for histopathological data
sets, regardless of well-known limitations in consistency for some tasks. To
date, some datasets on mitotic figures are available and were used for
development of promising deep learning-based algorithms. In order to assess
robustness of those algorithms and reproducibility of their methods it is
necessary to test on several independent datasets. The influence of different
labeling methods of these available datasets is currently unknown. To tackle
this, we present an alternative set of labels for the images of the auxiliary
mitosis dataset of the TUPAC16 challenge. Additional to manual mitotic figure
screening, we used a novel, algorithm-aided labeling process, that allowed to
minimize the risk of missing rare mitotic figures in the images. All potential
mitotic figures were independently assessed by two pathologists. The novel,
publicly available set of labels contains 1,999 mitotic figures (+28.80%) and
additionally includes 10,483 labels of cells with high similarities to mitotic
figures (hard examples). We found significant difference comparing F_1 scores
between the original label set (0.549) and the new alternative label set
(0.735) using a standard deep learning object detection architecture. The
models trained on the alternative set showed higher overall confidence values,
suggesting a higher overall label consistency. Findings of the present study
show that pathologists-defined labels may vary significantly resulting in
notable difference in the model performance. Comparison of deep learning-based
algorithms between independent datasets with different labeling methods should
be done with caution.
