Can recurrent neural networks warp time?
  Successful recurrent models such as long short-term memories (LSTMs) and
gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these
models have been found to improve the learning of medium to long term temporal
dependencies and to help with vanishing gradient issues. We prove that
learnable gates in a recurrent model formally provide quasi- invariance to
general time transformations in the input data. We recover part of the LSTM
architecture from a simple axiomatic approach. This result leads to a new way
of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new
chrono initialization is shown to greatly improve learning of long term
dependencies, with minimal implementation effort.
