Dilated Recurrent Neural Networks
  Learning with recurrent neural networks (RNNs) on long sequences is a
notoriously difficult task. There are three major challenges: 1) complex
dependencies, 2) vanishing and exploding gradients, and 3) efficient
parallelization. In this paper, we introduce a simple yet effective RNN
connection structure, the DilatedRNN, which simultaneously tackles all of these
challenges. The proposed architecture is characterized by multi-resolution
dilated recurrent skip connections and can be combined flexibly with diverse
RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and
enhances training efficiency significantly, while matching state-of-the-art
performance (even with standard RNN cells) in tasks involving very long-term
dependencies. To provide a theory-based quantification of the architecture's
advantages, we introduce a memory capacity measure, the mean recurrent length,
which is more suitable for RNNs with long skip connections than existing
measures. We rigorously prove the advantages of the DilatedRNN over other
recurrent neural architectures. The code for our method is publicly available
at https://github.com/code-terminator/DilatedRNN
