Generalized Bayesian Posterior Expectation Distillation for Deep Neural
  Networks
  In this paper, we present a general framework for distilling expectations
with respect to the Bayesian posterior distribution of a deep neural network
classifier, extending prior work on the Bayesian Dark Knowledge framework. The
proposed framework takes as input "teacher" and student model architectures and
a general posterior expectation of interest. The distillation method performs
an online compression of the selected posterior expectation using iteratively
generated Monte Carlo samples. We focus on the posterior predictive
distribution and expected entropy as distillation targets. We investigate
several aspects of this framework including the impact of uncertainty and the
choice of student model architecture. We study methods for student model
architecture search from a speed-storage-accuracy perspective and evaluate
down-stream tasks leveraging entropy distillation including uncertainty ranking
and out-of-distribution detection.
