Simplified Long Short-term Memory Recurrent Neural Networks: part II
  This is part II of three-part work. Here, we present a second set of
inter-related five variants of simplified Long Short-term Memory (LSTM)
recurrent neural networks by further reducing adaptive parameters. Two of these
models have been introduced in part I of this work. We evaluate and verify our
model variants on the benchmark MNIST dataset and assert that these models are
comparable to the base LSTM model while use progressively less number of
parameters. Moreover, we observe that in case of using the ReLU activation, the
test accuracy performance of the standard LSTM will drop after a number of
epochs when learning parameter become larger. However all of the new model
variants sustain their performance.
