Diversity Matters When Learning From Ensembles
  Deep ensembles excel in large-scale image classification tasks both in terms
of prediction accuracy and calibration. Despite being simple to train, the
computation and memory cost of deep ensembles limits their practicability.
While some recent works propose to distill an ensemble model into a single
model to reduce such costs, there is still a performance gap between the
ensemble and distilled models. We propose a simple approach for reducing this
gap, i.e., making the distilled performance close to the full ensemble. Our key
assumption is that a distilled model should absorb as much function diversity
inside the ensemble as possible. We first empirically show that the typical
distillation procedure does not effectively transfer such diversity, especially
for complex models that achieve near-zero training error. To fix this, we
propose a perturbation strategy for distillation that reveals diversity by
seeking inputs for which ensemble member outputs disagree. We empirically show
that a model distilled with such perturbed samples indeed exhibits enhanced
diversity, leading to improved performance.
