A Bayesian Approach to Recurrence in Neural Networks
  We begin by reiterating that common neural network activation functions have
simple Bayesian origins. In this spirit, we go on to show that Bayes's theorem
also implies a simple recurrence relation; this leads to a Bayesian recurrent
unit with a prescribed feedback formulation. We show that introduction of a
context indicator leads to a variable feedback that is similar to the forget
mechanism in conventional recurrent units. A similar approach leads to a
probabilistic input gate. The Bayesian formulation leads naturally to the two
pass algorithm of the Kalman smoother or forward-backward algorithm, meaning
that inference naturally depends upon future inputs as well as past ones.
Experiments on speech recognition confirm that the resulting architecture can
perform as well as a bidirectional recurrent network with the same number of
parameters as a unidirectional one. Further, when configured explicitly
bidirectionally, the architecture can exceed the performance of a conventional
bidirectional recurrence.
