Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation
  Knowledge distillation aims at obtaining a compact and effective model by
learning the mapping function from a much larger one. Due to the limited
capacity of the student, the student would underfit the teacher. Therefore,
student performance would unexpectedly drop when distilling from an oversized
teacher, termed the capacity gap problem. We investigate this problem by study
the gap of confidence between teacher and student. We find that the magnitude
of confidence is not necessary for knowledge distillation and could harm the
student performance if the student are forced to learn confidence. We propose
Spherical Knowledge Distillation to eliminate this gap explicitly, which eases
the underfitting problem. We find this novel knowledge representation can
improve compact models with much larger teachers and is robust to temperature.
We conducted experiments on both CIFAR100 and ImageNet, and achieve significant
improvement. Specifically, we train ResNet18 to 73.0 accuracy, which is a
substantial improvement over previous SOTA and is on par with resnet34 almost
twice the student size. The implementation has been shared at
https://github.com/forjiuzhou/Spherical-Knowledge-Distillation.
