Harmless label noise and informative soft-labels in supervised
  classification
  Manual labelling of training examples is common practice in supervised
learning. When the labelling task is of non-trivial difficulty, the supplied
labels may not be equal to the ground-truth labels, and label noise is
introduced into the training dataset. If the manual annotation is carried out
by multiple experts, the same training example can be given different class
assignments by different experts, which is indicative of label noise. In the
framework of model-based classification, a simple, but key observation is that
when the manual labels are sampled using the posterior probabilities of class
membership, the noisy labels are as valuable as the ground-truth labels in
terms of statistical information. A relaxation of this process is a random
effects model for imperfect labelling by a group that uses approximate
posterior probabilities of class membership. The relative efficiency of
logistic regression using the noisy labels compared to logistic regression
using the ground-truth labels can then be derived. The main finding is that
logistic regression can be robust to label noise when label noise and
classification difficulty are positively correlated. In particular, when
classification difficulty is the only source of label errors, multiple sets of
noisy labels can supply more information for the estimation of a classification
rule compared to the single set of ground-truth labels.
