Amalgamating Filtered Knowledge: Learning Task-customized Student from
  Multi-task Teachers
  Many well-trained Convolutional Neural Network(CNN) models have now been
released online by developers for the sake of effortless reproducing. In this
paper, we treat such pre-trained networks as teachers and explore how to learn
a target student network for customized tasks, using multiple teachers that
handle different tasks. We assume no human-labelled annotations are available,
and each teacher model can be either single- or multi-task network, where the
former is a degenerated case of the latter. The student model, depending on the
customized tasks, learns the related knowledge filtered from the multiple
teachers, and eventually masters the complete or a subset of expertise from all
teachers. To this end, we adopt a layer-wise training strategy, which entangles
the student's network block to be learned with the corresponding teachers. As
demonstrated on several benchmarks, the learned student network achieves very
promising results, even outperforming the teachers on the customized tasks.
