The Visual QA Devil in the Details: The Impact of Early Fusion and Batch
  Norm on CLEVR
  Visual QA is a pivotal challenge for higher-level reasoning, requiring
understanding language, vision, and relationships between many objects in a
scene. Although datasets like CLEVR are designed to be unsolvable without such
complex relational reasoning, some surprisingly simple feed-forward, "holistic"
models have recently shown strong performance on this dataset. These models
lack any kind of explicit iterative, symbolic reasoning procedure, which are
hypothesized to be necessary for counting objects, narrowing down the set of
relevant objects based on several attributes, etc. The reason for this strong
performance is poorly understood. Hence, our work analyzes such models, and
finds that minor architectural elements are crucial to performance. In
particular, we find that \textit{early fusion} of language and vision provides
large performance improvements. This contrasts with the late fusion approaches
popular at the dawn of Visual QA. We propose a simple module we call Multimodal
Core, which we hypothesize performs the fundamental operations for multimodal
tasks. We believe that understanding why these elements are so important to
complex question answering will aid the design of better-performing algorithms
for Visual QA while minimizing hand-engineering effort.
