Long Short-Term Memory as a Dynamically Computed Element-wise Weighted
  Sum
  LSTMs were introduced to combat vanishing gradients in simple RNNs by
augmenting them with gated additive recurrent connections. We present an
alternative view to explain the success of LSTMs: the gates themselves are
versatile recurrent models that provide more representational power than
previously appreciated. We do this by decoupling the LSTM's gates from the
embedded simple RNN, producing a new class of RNNs where the recurrence
computes an element-wise weighted sum of context-independent functions of the
input. Ablations on a range of problems demonstrate that the gating mechanism
alone performs as well as an LSTM in most settings, strongly suggesting that
the gates are doing much more in practice than just alleviating vanishing
gradients.
