When Does Label Smoothing Help?
  The generalization and learning speed of a multi-class neural network can
often be significantly improved by using soft targets that are a weighted
average of the hard targets and the uniform distribution over labels. Smoothing
the labels in this way prevents the network from becoming over-confident and
label smoothing has been used in many state-of-the-art models, including image
classification, language translation and speech recognition. Despite its
widespread use, label smoothing is still poorly understood. Here we show
empirically that in addition to improving generalization, label smoothing
improves model calibration which can significantly improve beam-search.
However, we also observe that if a teacher network is trained with label
smoothing, knowledge distillation into a student network is much less
effective. To explain these observations, we visualize how label smoothing
changes the representations learned by the penultimate layer of the network. We
show that label smoothing encourages the representations of training examples
from the same class to group in tight clusters. This results in loss of
information in the logits about resemblances between instances of different
classes, which is necessary for distillation, but does not hurt generalization
or calibration of the model's predictions.
