Knowledge Flow: Improve Upon Your Teachers
  A zoo of deep nets is available these days for almost any given task, and it
is increasingly unclear which net to start with when addressing a new task, or
which net to use as an initialization for fine-tuning a new model. To address
this issue, in this paper, we develop knowledge flow which moves 'knowledge'
from multiple deep nets, referred to as teachers, to a new deep net model,
called the student. The structure of the teachers and the student can differ
arbitrarily and they can be trained on entirely different tasks with different
output spaces too. Upon training with knowledge flow the student is independent
of the teachers. We demonstrate our approach on a variety of supervised and
reinforcement learning tasks, outperforming fine-tuning and other 'knowledge
exchange' methods.
