Improving the Gating Mechanism of Recurrent Neural Networks
  Gating mechanisms are widely used in neural network models, where they allow
gradients to backpropagate more easily through depth or time. However, their
saturation property introduces problems of its own. For example, in recurrent
models these gates need to have outputs near 1 to propagate information over
long time-delays, which requires them to operate in their saturation regime and
hinders gradient-based learning of the gate mechanism. We address this problem
by deriving two synergistic modifications to the standard gating mechanism that
are easy to implement, introduce no additional hyperparameters, and improve
learnability of the gates when they are close to saturation. We show how these
changes are related to and improve on alternative recently proposed gating
mechanisms such as chrono initialization and Ordered Neurons. Empirically, our
simple gating mechanisms robustly improve the performance of recurrent models
on a range of applications, including synthetic memorization tasks, sequential
image classification, language modeling, and reinforcement learning,
particularly when long-term dependencies are involved.
