Modelling Identity Rules with Neural Networks
  In this paper, we show that standard feed-forward and recurrent neural
networks fail to learn abstract patterns based on identity rules. We propose
Relation Based Pattern (RBP) extensions to neural network structures that solve
this problem and answer, as well as raise, questions about integrating
structures for inductive bias into neural networks. Examples of abstract
patterns are the sequence patterns ABA and ABB where A or B can be any object.
These were introduced by Marcus et al (1999) who also found that 7 month old
infants recognise these patterns in sequences that use an unfamiliar vocabulary
while simple recurrent neural networks do not.This result has been contested in
the literature but it is confirmed by our experiments. We also show that the
inability to generalise extends to different, previously untested, settings. We
propose a new approach to modify standard neural network architectures, called
Relation Based Patterns (RBP) with different variants for classification and
prediction. Our experiments show that neural networks with the appropriate RBP
structure achieve perfect classification and prediction performance on
synthetic data, including mixed concrete and abstract patterns. RBP also
improves neural network performance in experiments with real-world sequence
prediction tasks. We discuss these finding in terms of challenges for neural
network models and identify consequences from this result in terms of
developing inductive biases for neural network learning.
