Differentiable Reasoning over Long Stories -- Assessing Systematic
  Generalisation in Neural Models
  Contemporary neural networks have achieved a series of developments and
successes in many aspects; however, when exposed to data outside the training
distribution, they may fail to predict correct answers. In this work, we were
concerned about this generalisation issue and thus analysed a broad set of
models systematically and robustly over long stories. Related experiments were
conducted based on the CLUTRR, which is a diagnostic benchmark suite that can
analyse generalisation of natural language understanding (NLU) systems by
training over small story graphs and testing on larger ones. In order to handle
the multi-relational story graph, we consider two classes of neural models:
"E-GNN", the graph-based models that can process graph-structured data and
consider the edge attributes simultaneously; and "L-Graph", the sequence-based
models which can process linearized version of the graphs. We performed an
extensive empirical evaluation, and we found that the modified recurrent neural
network yield surprisingly accurate results across every systematic
generalisation tasks which outperform the modified graph neural network, while
the latter produced more robust models.
