Knowledge distillation via adaptive instance normalization
  This paper addresses the problem of model compression via knowledge
distillation. To this end, we propose a new knowledge distillation method based
on transferring feature statistics, specifically the channel-wise mean and
variance, from the teacher to the student. Our method goes beyond the standard
way of enforcing the mean and variance of the student to be similar to those of
the teacher through an $L_2$ loss, which we found it to be of limited
effectiveness. Specifically, we propose a new loss based on adaptive instance
normalization to effectively transfer the feature statistics. The main idea is
to transfer the learned statistics back to the teacher via adaptive instance
normalization (conditioned on the student) and let the teacher network
"evaluate" via a loss whether the statistics learned by the student are
reliably transferred. We show that our distillation method outperforms other
state-of-the-art distillation methods over a large set of experimental settings
including different (a) network architectures, (b) teacher-student capacities,
(c) datasets, and (d) domains.
