Human-centric Metric for Accelerating Pathology Reports Annotation
  Pathology reports contain useful information such as the main involved organ,
diagnosis, etc. These information can be identified from the free text reports
and used for large-scale statistical analysis or serve as annotation for other
modalities such as pathology slides images. However, manual classification for
a huge number of reports on multiple tasks is labor-intensive. In this paper,
we have developed an automatic text classifier based on BERT and we propose a
human-centric metric to evaluate the model. According to the model confidence,
we identify low-confidence cases that require further expert annotation and
high-confidence cases that are automatically classified. We report the
percentage of low-confidence cases and the performance of automatically
classified cases. On the high-confidence cases, the model achieves
classification accuracy comparable to pathologists. This leads a potential of
reducing 80% to 98% of the manual annotation workload.
