Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
  The recurrent network architecture is a widely used model in sequence
modeling, but its serial dependency hinders the computation parallelization,
which makes the operation inefficient. The same problem was encountered in
serial adder at the early stage of digital electronics. In this paper, we
discuss the similarities between recurrent neural network (RNN) and serial
adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module
to RNN, which makes it possible for RNN to run in parallel. Then, we design the
method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is
proposed. CL-RNN takes advantages in parallelism and flexible receptive field.
Through a comprehensive set of tests, we verify that CL-RNN can perform better
than existing typical RNNs in sequence modeling tasks which are specially
designed for RNNs.
