Learning Various Length Dependence by Dual Recurrent Neural Networks
  Recurrent neural networks (RNNs) are widely used as a memory model for
sequence-related problems. Many variants of RNN have been proposed to solve the
gradient problems of training RNNs and process long sequences. Although some
classical models have been proposed, capturing long-term dependence while
responding to short-term changes remains a challenge. To this problem, we
propose a new model named Dual Recurrent Neural Networks (DuRNN). The DuRNN
consists of two parts to learn the short-term dependence and progressively
learn the long-term dependence. The first part is a recurrent neural network
with constrained full recurrent connections to deal with short-term dependence
in sequence and generate short-term memory. Another part is a recurrent neural
network with independent recurrent connections which helps to learn long-term
dependence and generate long-term memory. A selection mechanism is added
between two parts to help the needed long-term information transfer to the
independent neurons. Multiple modules can be stacked to form a multi-layer
model for better performance. Our contributions are: 1) a new recurrent model
developed based on the divide-and-conquer strategy to learn long and short-term
dependence separately, and 2) a selection mechanism to enhance the separating
and learning of different temporal scales of dependence. Both theoretical
analysis and extensive experiments are conducted to validate the performance of
our model, and we also conduct simple visualization experiments and ablation
analyses for the model interpretability. Experimental results indicate that the
proposed DuRNN model can handle not only very long sequences (over 5000 time
steps), but also short sequences very well. Compared with many state-of-the-art
RNN models, our model has demonstrated efficient and better performance.
