Variable Computation in Recurrent Neural Networks
  Recurrent neural networks (RNNs) have been used extensively and with
increasing success to model various types of sequential data. Much of this
progress has been achieved through devising recurrent units and architectures
with the flexibility to capture complex statistics in the data, such as long
range dependency or localized attention phenomena. However, while many
sequential data (such as video, speech or language) can have highly variable
information flow, most recurrent models still consume input features at a
constant rate and perform a constant number of computations per time step,
which can be detrimental to both speed and model capacity. In this paper, we
explore a modification to existing recurrent units which allows them to learn
to vary the amount of computation they perform at each step, without prior
knowledge of the sequence's time structure. We show experimentally that not
only do our models require fewer operations, they also lead to better
performance overall on evaluation tasks.
