Prediction-Guided Distillation for Dense Object Detection
  Real-world object detection models should be cheap and accurate. Knowledge
distillation (KD) can boost the accuracy of a small, cheap detection model by
leveraging useful information from a larger teacher model. However, a key
challenge is identifying the most informative features produced by the teacher
for distillation. In this work, we show that only a very small fraction of
features within a ground-truth bounding box are responsible for a teacher's
high detection performance. Based on this, we propose Prediction-Guided
Distillation (PGD), which focuses distillation on these key predictive regions
of the teacher and yields considerable gains in performance over many existing
KD baselines. In addition, we propose an adaptive weighting scheme over the key
regions to smooth out their influence and achieve even better performance. Our
proposed approach outperforms current state-of-the-art KD baselines on a
variety of advanced one-stage detection architectures. Specifically, on the
COCO dataset, our method achieves between +3.1% and +4.6% AP improvement using
ResNet-101 and ResNet-50 as the teacher and student backbones, respectively. On
the CrowdHuman dataset, we achieve +3.2% and +2.0% improvements in MR and AP,
also using these backbones. Our code is available at
https://github.com/ChenhongyiYang/PGD.
