Visual Relationship Detection with Visual-Linguistic Knowledge from
  Multimodal Representations
  Visual relationship detection aims to reason over relationships among salient
objects in images, which has drawn increasing attention over the past few
years. Inspired by human reasoning mechanisms, it is believed that external
visual commonsense knowledge is beneficial for reasoning visual relationships
of objects in images, which is however rarely considered in existing methods.
In this paper, we propose a novel approach named Relational Visual-Linguistic
Bidirectional Encoder Representations from Transformers (RVL-BERT), which
performs relational reasoning with both visual and language commonsense
knowledge learned via self-supervised pre-training with multimodal
representations. RVL-BERT also uses an effective spatial module and a novel
mask attention module to explicitly capture spatial information among the
objects. Moreover, our model decouples object detection from visual
relationship recognition by taking in object names directly, enabling it to be
used on top of any object detection system. We show through quantitative and
qualitative experiments that, with the transferred knowledge and novel modules,
RVL-BERT achieves competitive results on two challenging visual relationship
detection datasets. The source code is available at
https://github.com/coldmanck/RVL-BERT.
