SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
  Existing VQA datasets contain questions with varying levels of complexity.
While the majority of questions in these datasets require perception for
recognizing existence, properties, and spatial relationships of entities, a
significant portion of questions pose challenges that correspond to reasoning
tasks - tasks that can only be answered through a synthesis of perception and
knowledge about the world, logic and / or reasoning. Analyzing performance
across this distinction allows us to notice when existing VQA models have
consistency issues; they answer the reasoning questions correctly but fail on
associated low-level perception questions. For example, in Figure 1, models
answer the complex reasoning question "Is the banana ripe enough to eat?"
correctly, but fail on the associated perception question "Are the bananas
mostly green or yellow?" indicating that the model likely answered the
reasoning question correctly but for the wrong reason. We quantify the extent
to which this phenomenon occurs by creating a new Reasoning split of the VQA
dataset and collecting VQA-introspect, a new dataset1 which consists of 238K
new perception questions which serve as sub questions corresponding to the set
of perceptual tasks needed to effectively answer the complex reasoning
questions in the Reasoning split. Our evaluation shows that state-of-the-art
VQA models have comparable performance in answering perception and reasoning
questions, but suffer from consistency problems. To address this shortcoming,
we propose an approach called Sub-Question Importance-aware Network Tuning
(SQuINT), which encourages the model to attend to the same parts of the image
when answering the reasoning question and the perception sub question. We show
that SQuINT improves model consistency by ~5%, also marginally improving
performance on the Reasoning questions in VQA, while also displaying better
attention maps.
