R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual
  Question Answering
  Recently, Visual Question Answering (VQA) has emerged as one of the most
significant tasks in multimodal learning as it requires understanding both
visual and textual modalities. Existing methods mainly rely on extracting image
and question features to learn their joint feature embedding via multimodal
fusion or attention mechanism. Some recent studies utilize external
VQA-independent models to detect candidate entities or attributes in images,
which serve as semantic knowledge complementary to the VQA task. However, these
candidate entities or attributes might be unrelated to the VQA task and have
limited semantic capacities. To better utilize semantic knowledge in images, we
propose a novel framework to learn visual relation facts for VQA. Specifically,
we build up a Relation-VQA (R-VQA) dataset based on the Visual Genome dataset
via a semantic similarity module, in which each data consists of an image, a
corresponding question, a correct answer and a supporting relation fact. A
well-defined relation detector is then adopted to predict visual
question-related relation facts. We further propose a multi-step attention
model composed of visual attention and semantic attention sequentially to
extract related visual knowledge and semantic knowledge. We conduct
comprehensive experiments on the two benchmark datasets, demonstrating that our
model achieves state-of-the-art performance and verifying the benefit of
considering visual relation facts.
