Feed-Forward Neural Networks Need Inductive Bias to Learn Equality
  Relations
  Basic binary relations such as equality and inequality are fundamental to
relational data structures. Neural networks should learn such relations and
generalise to new unseen data. We show in this study, however, that this
generalisation fails with standard feed-forward networks on binary vectors.
Even when trained with maximal training data, standard networks do not reliably
detect equality.We introduce differential rectifier (DR) units that we add to
the network in different configurations. The DR units create an inductive bias
in the networks, so that they do learn to generalise, even from small numbers
of examples and we have not found any negative effect of their inclusion in the
network. Given the fundamental nature of these relations, we hypothesize that
feed-forward neural network learning benefits from inductive bias in other
relations as well. Consequently, the further development of suitable inductive
biases will be beneficial to many tasks in relational learning with neural
networks.
