Gated Orthogonal Recurrent Units: On Learning to Forget
  We present a novel recurrent neural network (RNN) based model that combines
the remembering ability of unitary RNNs with the ability of gated RNNs to
effectively forget redundant/irrelevant information in its memory. We achieve
this by extending unitary RNNs with a gating mechanism. Our model is able to
outperform LSTMs, GRUs and Unitary RNNs on several long-term dependency
benchmark tasks. We empirically both show the orthogonal/unitary RNNs lack the
ability to forget and also the ability of GORU to simultaneously remember long
term dependencies while forgetting irrelevant information. This plays an
important role in recurrent neural networks. We provide competitive results
along with an analysis of our model on many natural sequential tasks including
the bAbI Question Answering, TIMIT speech spectrum prediction, Penn TreeBank,
and synthetic tasks that involve long-term dependencies such as algorithmic,
parenthesis, denoising and copying tasks.
