Supervising the Transfer of Reasoning Patterns in VQA
  Methods for Visual Question Anwering (VQA) are notorious for leveraging
dataset biases rather than performing reasoning, hindering generalization. It
has been recently shown that better reasoning patterns emerge in attention
layers of a state-of-the-art VQA model when they are trained on perfect
(oracle) visual inputs. This provides evidence that deep neural networks can
learn to reason when training conditions are favorable enough. However,
transferring this learned knowledge to deployable models is a challenge, as
much of it is lost during the transfer. We propose a method for knowledge
transfer based on a regularization term in our loss function, supervising the
sequence of required reasoning operations. We provide a theoretical analysis
based on PAC-learning, showing that such program prediction can lead to
decreased sample complexity under mild hypotheses. We also demonstrate the
effectiveness of this approach experimentally on the GQA dataset and show its
complementarity to BERT-like self-supervised pre-training.
