Representation Transfer by Optimal Transport
  Learning generic representations with deep networks requires massive training
samples and significant computer resources. To learn a new specific task, an
important issue is to transfer the generic teacher's representation to a
student network. In this paper, we propose to use a metric between
representations that is based on a functional view of neurons. We use optimal
transport to quantify the match between two representations, yielding a
distance that embeds some invariances inherent to the representation of deep
networks. This distance defines a regularizer promoting the similarity of the
student's representation with that of the teacher. Our approach can be used in
any learning context where representation transfer is applicable. We experiment
here on two standard settings: inductive transfer learning, where the teacher's
representation is transferred to a student network of same architecture for a
new related task, and knowledge distillation, where the teacher's
representation is transferred to a student of simpler architecture for the same
task (model compression). Our approach also lends itself to solving new
learning problems; we demonstrate this by showing how to directly transfer the
teacher's representation to a simpler architecture student for a new related
task.
