Can non-specialists provide high quality gold standard labels in
  challenging modalities?
  Probably yes. -- Supervised Deep Learning dominates performance scores for
many computer vision tasks and defines the state-of-the-art. However, medical
image analysis lags behind natural image applications. One of the many reasons
is the lack of well annotated medical image data available to researchers. One
of the first things researchers are told is that we require significant
expertise to reliably and accurately interpret and label such data. We see
significant inter- and intra-observer variability between expert annotations of
medical images. Still, it is a widely held assumption that novice annotators
are unable to provide useful annotations for use by clinical Deep Learning
models. In this work we challenge this assumption and examine the implications
of using a minimally trained novice labelling workforce to acquire annotations
for a complex medical image dataset. We study the time and cost implications of
using novice annotators, the raw performance of novice annotators compared to
gold-standard expert annotators, and the downstream effects on a trained Deep
Learning segmentation model's performance for detecting a specific congenital
heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.
