Knowledge Distillation and Student-Teacher Learning for Visual
  Intelligence: A Review and New Outlooks
  Deep neural models in recent years have been successful in almost every
field, including extremely complex problem statements. However, these models
are huge in size, with millions (and even billions) of parameters, thus
demanding more heavy computation power and failing to be deployed on edge
devices. Besides, the performance boost is highly dependent on redundant
labeled data. To achieve faster speeds and to handle the problems caused by the
lack of data, knowledge distillation (KD) has been proposed to transfer
information learned from one model to another. KD is often characterized by the
so-called `Student-Teacher' (S-T) learning framework and has been broadly
applied in model compression and knowledge transfer. This paper is about KD and
S-T learning, which are being actively studied in recent years. First, we aim
to provide explanations of what KD is and how/why it works. Then, we provide a
comprehensive survey on the recent progress of KD methods together with S-T
frameworks typically for vision tasks. In general, we consider some fundamental
questions that have been driving this research area and thoroughly generalize
the research progress and technical details. Additionally, we systematically
analyze the research status of KD in vision applications. Finally, we discuss
the potentials and open challenges of existing methods and prospect the future
directions of KD and S-T learning.
