SOrT-ing VQA Models : Contrastive Gradient Learning for Improved
  Consistency
  Recent research in Visual Question Answering (VQA) has revealed
state-of-the-art models to be inconsistent in their understanding of the world
-- they answer seemingly difficult questions requiring reasoning correctly but
get simpler associated sub-questions wrong. These sub-questions pertain to
lower level visual concepts in the image that models ideally should understand
to be able to answer the higher level question correctly. To address this, we
first present a gradient-based interpretability approach to determine the
questions most strongly correlated with the reasoning question on an image, and
use this to evaluate VQA models on their ability to identify the relevant
sub-questions needed to answer a reasoning question. Next, we propose a
contrastive gradient learning based approach called Sub-question Oriented
Tuning (SOrT) which encourages models to rank relevant sub-questions higher
than irrelevant questions for an <image, reasoning-question> pair. We show that
SOrT improves model consistency by upto 6.5% points over existing baselines,
while also improving visual grounding.
