Gated Feedback Recurrent Neural Networks
  In this work, we propose a novel recurrent neural network (RNN) architecture.
The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of
stacking multiple recurrent layers by allowing and controlling signals flowing
from upper recurrent layers to lower layers using a global gating unit for each
pair of layers. The recurrent signals exchanged between layers are gated
adaptively based on the previous hidden states and the current input. We
evaluated the proposed GF-RNN with different types of recurrent units, such as
tanh, long short-term memory and gated recurrent units, on the tasks of
character-level language modeling and Python program evaluation. Our empirical
evaluation of different RNN units, revealed that in both tasks, the GF-RNN
outperforms the conventional approaches to build deep stacked RNNs. We suggest
that the improvement arises because the GF-RNN can adaptively assign different
layers to different timescales and layer-to-layer interactions (including the
top-down ones which are not usually present in a stacked RNN) by learning to
gate these interactions.
