Is Label Smoothing Truly Incompatible with Knowledge Distillation: An
  Empirical Study
  This work aims to empirically clarify a recently discovered perspective that
label smoothing is incompatible with knowledge distillation. We begin by
introducing the motivation behind on how this incompatibility is raised, i.e.,
label smoothing erases relative information between teacher logits. We provide
a novel connection on how label smoothing affects distributions of semantically
similar and dissimilar classes. Then we propose a metric to quantitatively
measure the degree of erased information in sample's representation. After
that, we study its one-sidedness and imperfection of the incompatibility view
through massive analyses, visualizations and comprehensive experiments on Image
Classification, Binary Networks, and Neural Machine Translation. Finally, we
broadly discuss several circumstances wherein label smoothing will indeed lose
its effectiveness. Project page:
http://zhiqiangshen.com/projects/LS_and_KD/index.html.
