Deep Independently Recurrent Neural Network (IndRNN)
  Recurrent neural networks (RNNs) are known to be difficult to train due to
the gradient vanishing and exploding problems and thus difficult to learn
long-term patterns and construct deep networks. To address these problems, this
paper proposes a new type of RNNs with the recurrent connection formulated as
Hadamard product, referred to as independently recurrent neural network
(IndRNN), where neurons in the same layer are independent of each other and
connected across layers. Due to the better behaved gradient backpropagation,
IndRNN with regulated recurrent weights effectively addresses the gradient
vanishing and exploding problems and thus long-term dependencies can be
learned. Moreover, an IndRNN can work with non-saturated activation functions
such as ReLU (rectified linear unit) and be still trained robustly. Different
deeper IndRNN architectures, including the basic stacked IndRNN, residual
IndRNN and densely connected IndRNN, have been investigated, all of which can
be much deeper than the existing RNNs. Furthermore, IndRNN reduces the
computation at each time step and can be over 10 times faster than the commonly
used Long short-term memory (LSTM). Experimental results have shown that the
proposed IndRNN is able to process very long sequences and construct very deep
networks. Better performance has been achieved on various tasks with IndRNNs
compared with the traditional RNN, LSTM and the popular Transformer.
