Customizing Student Networks From Heterogeneous Teachers via Adaptive
  Knowledge Amalgamation
  A massive number of well-trained deep networks have been released by
developers online. These networks may focus on different tasks and in many
cases are optimized for different datasets. In this paper, we study how to
exploit such heterogeneous pre-trained networks, known as teachers, so as to
train a customized student network that tackles a set of selective tasks
defined by the user. We assume no human annotations are available, and each
teacher may be either single- or multi-task. To this end, we introduce a
dual-step strategy that first extracts the task-specific knowledge from the
heterogeneous teachers sharing the same sub-task, and then amalgamates the
extracted knowledge to build the student network. To facilitate the training,
we employ a selective learning scheme where, for each unlabelled sample, the
student learns adaptively from only the teacher with the least prediction
ambiguity. We evaluate the proposed approach on several datasets and
experimental results demonstrate that the student, learned by such adaptive
knowledge amalgamation, achieves performances even better than those of the
teachers.
