Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance
  Tradeoff Perspective
  Knowledge distillation is an effective approach to leverage a well-trained
network or an ensemble of them, named as the teacher, to guide the training of
a student network. The outputs from the teacher network are used as soft labels
for supervising the training of a new network. Recent studies
\citep{muller2019does,yuan2020revisiting} revealed an intriguing property of
the soft labels that making labels soft serves as a good regularization to the
student network. From the perspective of statistical learning, regularization
aims to reduce the variance, however how bias and variance change is not clear
for training with soft labels. In this paper, we investigate the bias-variance
tradeoff brought by distillation with soft labels. Specifically, we observe
that during training the bias-variance tradeoff varies sample-wisely. Further,
under the same distillation temperature setting, we observe that the
distillation performance is negatively associated with the number of some
specific samples, which are named as regularization samples since these samples
lead to bias increasing and variance decreasing. Nevertheless, we empirically
find that completely filtering out regularization samples also deteriorates
distillation performance. Our discoveries inspired us to propose the novel
weighted soft labels to help the network adaptively handle the sample-wise
bias-variance tradeoff. Experiments on standard evaluation benchmarks validate
the effectiveness of our method. Our code is available at
\url{https://github.com/bellymonster/Weighted-Soft-Label-Distillation}.
