Open-vocabulary Object Detection via Vision and Language Knowledge
  Distillation
  We aim at advancing open-vocabulary object detection, which detects objects
described by arbitrary text inputs. The fundamental challenge is the
availability of training data. It is costly to further scale up the number of
classes contained in existing object detection datasets. To overcome this
challenge, we propose ViLD, a training method via Vision and Language knowledge
Distillation. Our method distills the knowledge from a pretrained
open-vocabulary image classification model (teacher) into a two-stage detector
(student). Specifically, we use the teacher model to encode category texts and
image regions of object proposals. Then we train a student detector, whose
region embeddings of detected boxes are aligned with the text and image
embeddings inferred by the teacher. We benchmark on LVIS by holding out all
rare categories as novel categories that are not seen during training. ViLD
obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the
supervised counterpart by 3.8. When trained with a stronger teacher model
ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other
datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on
COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous
state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are
open-sourced at
https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.
