Distillation from heterogeneous unlabeled collections
  Compressing deep networks is essential to expand their range of applications
to constrained settings. The need for compression however often arises long
after the model was trained, when the original data might no longer be
available. On the other hand, unlabeled data, not necessarily related to the
target task, is usually plentiful, especially in image classification tasks. In
this work, we propose a scheme to leverage such samples to distill the
knowledge learned by a large teacher network to a smaller student. The proposed
technique relies on (i) preferentially sampling datapoints that appear related,
and (ii) taking better advantage of the learning signal. We show that the
former speeds up the student's convergence, while the latter boosts its
performance, achieving performances closed to what can be expected with the
original data.
