Generalized Knowledge Distillation via Relationship Matching
The knowledge of a well-trained deep neural network (a.k.a. the "teacher") is
valuable for learning similar tasks. Knowledge distillation extracts knowledge
from the teacher and integrates it with the target model (a.k.a. the
"student"), which expands the student's knowledge and improves its learning
efficacy. Instead of enforcing the teacher to work on the same task as the
student, we borrow the knowledge from a teacher trained from a general label
space -- in this "Generalized Knowledge Distillation (GKD)", the classes of the
teacher and the student may be the same, completely different, or partially
overlapped. We claim that the comparison ability between instances acts as an
essential factor threading knowledge across tasks, and propose the RElationship
FacIlitated Local cLassifiEr Distillation (REFILLED) approach, which decouples
the GKD flow of the embedding and the top-layer classifier. In particular,
different from reconciling the instance-label confidence between models,
REFILLED requires the teacher to reweight the hard tuples pushed forward by the
student and then matches the similarity comparison levels between instances. An
embedding-induced classifier based on the teacher model supervises the
student's classification confidence and adaptively emphasizes the most related
supervision from the teacher. REFILLED demonstrates strong discriminative
ability when the classes of the teacher vary from the same to a fully
non-overlapped set w.r.t. the student. It also achieves state-of-the-art
performance on standard knowledge distillation, one-step incremental learning,
and few-shot learning tasks.