Better Supervisory Signals by Observing Learning Paths
  Better-supervised models might have better performance. In this paper, we
first clarify what makes for good supervision for a classification problem, and
then explain two existing label refining methods, label smoothing and knowledge
distillation, in terms of our proposed criterion. To further answer why and how
better supervision emerges, we observe the learning path, i.e., the trajectory
of the model's predictions during training, for each training sample. We find
that the model can spontaneously refine "bad" labels through a "zig-zag"
learning path, which occurs on both toy and real datasets. Observing the
learning path not only provides a new perspective for understanding knowledge
distillation, overfitting, and learning dynamics, but also reveals that the
supervisory signal of a teacher network can be very unstable near the best
points in training on real tasks. Inspired by this, we propose a new knowledge
distillation scheme, Filter-KD, which improves downstream classification
performance in various settings.
