Weight Priors for Learning Identity Relations
  Learning abstract and systematic relations has been an open issue in neural
network learning for over 30 years. It has been shown recently that neural
networks do not learn relations based on identity and are unable to generalize
well to unseen data. The Relation Based Pattern (RBP) approach has been
proposed as a solution for this problem. In this work, we extend RBP by
realizing it as a Bayesian prior on network weights to model the identity
relations. This weight prior leads to a modified regularization term in
otherwise standard network learning. In our experiments, we show that the
Bayesian weight priors lead to perfect generalization when learning identity
based relations and do not impede general neural network learning. We believe
that the approach of creating an inductive bias with weight priors can be
extended easily to other forms of relations and will be beneficial for many
other learning tasks.
