Confidence-Aware Multi-Teacher Knowledge Distillation
  Knowledge distillation is initially introduced to utilize additional
supervision from a single teacher model for the student model training. To
boost the student performance, some recent variants attempt to exploit diverse
knowledge sources from multiple teachers. However, existing studies mainly
integrate knowledge from diverse sources by averaging over multiple teacher
predictions or combining them using other various label-free strategies, which
may mislead student in the presence of low-quality teacher predictions. To
tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge
Distillation (CA-MKD), which adaptively assigns sample-wise reliability for
each teacher prediction with the help of ground-truth labels, with those
teacher predictions close to one-hot labels assigned large weights. Besides,
CA-MKD incorporates intermediate layers to stable the knowledge transfer
process. Extensive experiments show that our CA-MKD consistently outperforms
all compared state-of-the-art methods across various teacher-student
architectures.
