SLIM LSTMs
  Long Short-Term Memory (LSTM) Recurrent Neural networks (RNNs) rely on gating
signals, each driven by a function of a weighted sum of at least 3 components:
(i) one of an adaptive weight matrix multiplied by the incoming external input
vector sequence, (ii) one adaptive weight matrix multiplied by the previous
memory/state vector, and (iii) one adaptive bias vector. In effect, they
augment the simple Recurrent Neural Networks (sRNNs) structure with the
addition of a "memory cell" and the incorporation of at most 3 gating signals.
  The standard LSTM structure and components encompass redundancy and overly
increased parameterization. In this paper, we systemically introduce variants
of the LSTM RNNs, referred to as SLIM LSTMs. These variants express
aggressively reduced parameterizations to achieve computational saving and/or
speedup in (training) performance---while necessarily retaining (validation
accuracy) performance comparable to the standard LSTM RNN.
