How Modular Should Neural Module Networks Be for Systematic
  Generalization?
  Neural Module Networks (NMNs) aim at Visual Question Answering (VQA) via
composition of modules that tackle a sub-task. NMNs are a promising strategy to
achieve systematic generalization, i.e., overcoming biasing factors in the
training distribution. However, the aspects of NMNs that facilitate systematic
generalization are not fully understood. In this paper, we demonstrate that the
degree of modularity of the NMN have large influence on systematic
generalization. In a series of experiments on three VQA datasets (VQA-MNIST,
SQOOP, and CLEVR-CoGenT), our results reveal that tuning the degree of
modularity, especially at the image encoder stage, reaches substantially higher
systematic generalization. These findings lead to new NMN architectures that
outperform previous ones in terms of systematic generalization.
