Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  Large-scale pre-training methods of learning cross-modal representations on
image-text pairs are becoming popular for vision-language tasks. While existing
methods simply concatenate image region features and text features as input to
the model to be pre-trained and use self-attention to learn image-text semantic
alignments in a brute force manner, in this paper, we propose a new learning
method Oscar (Object-Semantics Aligned Pre-training), which uses object tags
detected in images as anchor points to significantly ease the learning of
alignments. Our method is motivated by the observation that the salient objects
in an image can be accurately detected, and are often mentioned in the paired
text. We pre-train an Oscar model on the public corpus of 6.5 million
text-image pairs, and fine-tune it on downstream tasks, creating new
state-of-the-arts on six well-established vision-language understanding and
generation tasks.
