VinVL: Revisiting Visual Representations in Vision-Language Models
  This paper presents a detailed study of improving visual representations for
vision language (VL) tasks and develops an improved object detection model to
provide object-centric representations of images. Compared to the most widely
used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new
model is bigger, better-designed for VL tasks, and pre-trained on much larger
training corpora that combine multiple public annotated object detection
datasets. Therefore, it can generate representations of a richer collection of
visual objects and concepts. While previous VL research focuses mainly on
improving the vision-language fusion model and leaves the object detection
model improvement untouched, we show that visual features matter significantly
in VL models. In our experiments we feed the visual features generated by the
new object detection model into a Transformer-based VL fusion model \oscar
\cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the
VL model and fine-tune it on a wide range of downstream VL tasks. Our results
show that the new visual features significantly improve the performance across
all VL tasks, creating new state-of-the-art results on seven public benchmarks.
We will release the new object detection model to public.
