Self-Distillation as Instance-Specific Label Smoothing
  It has been recently demonstrated that multi-generational self-distillation
can improve generalization. Despite this intriguing observation, reasons for
the enhancement remain poorly understood. In this paper, we first demonstrate
experimentally that the improved performance of multi-generational
self-distillation is in part associated with the increasing diversity in
teacher predictions. With this in mind, we offer a new interpretation for
teacher-student training as amortized MAP estimation, such that teacher
predictions enable instance-specific regularization. Our framework allows us to
theoretically relate self-distillation to label smoothing, a commonly used
technique that regularizes predictive uncertainty, and suggests the importance
of predictive diversity in addition to predictive uncertainty. We present
experimental results using multiple datasets and neural network architectures
that, overall, demonstrate the utility of predictive diversity. Finally, we
propose a novel instance-specific label smoothing technique that promotes
predictive diversity without the need for a separately trained teacher model.
We provide an empirical evaluation of the proposed method, which, we find,
often outperforms classical label smoothing.
