Augmenting Visual Question Answering with Semantic Frame Information in
  a Multitask Learning Approach
  Visual Question Answering (VQA) concerns providing answers to Natural
Language questions about images. Several deep neural network approaches have
been proposed to model the task in an end-to-end fashion. Whereas the task is
grounded in visual processing, if the question focuses on events described by
verbs, the language understanding component becomes crucial. Our hypothesis is
that models should be aware of verb semantics, as expressed via semantic role
labels, argument types, and/or frame elements. Unfortunately, no VQA dataset
exists that includes verb semantic information. Our first contribution is a new
VQA dataset (imSituVQA) that we built by taking advantage of the imSitu
annotations. The imSitu dataset consists of images manually labeled with
semantic frame elements, mostly taken from FrameNet. Second, we propose a
multitask CNN-LSTM VQA model that learns to classify the answers as well as the
semantic frame elements. Our experiments show that semantic frame element
classification helps the VQA system avoid inconsistent responses and improves
performance.
