Knowledge Distillation as Semiparametric Inference
  A popular approach to model compression is to train an inexpensive student
model to mimic the class probabilities of a highly accurate but cumbersome
teacher model. Surprisingly, this two-step knowledge distillation process often
leads to higher accuracy than training the student directly on labeled data. To
explain and enhance this phenomenon, we cast knowledge distillation as a
semiparametric inference problem with the optimal student model as the target,
the unknown Bayes class probabilities as nuisance, and the teacher
probabilities as a plug-in nuisance estimate. By adapting modern semiparametric
tools, we derive new guarantees for the prediction error of standard
distillation and develop two enhancements -- cross-fitting and loss correction
-- to mitigate the impact of teacher overfitting and underfitting on student
performance. We validate our findings empirically on both tabular and image
data and observe consistent improvements from our knowledge distillation
enhancements.
