Assessing the Memory Ability of Recurrent Neural Networks
  It is known that Recurrent Neural Networks (RNNs) can remember, in their
hidden layers, part of the semantic information expressed by a sequence (e.g.,
a sentence) that is being processed. Different types of recurrent units have
been designed to enable RNNs to remember information over longer time spans.
However, the memory abilities of different recurrent units are still
theoretically and empirically unclear, thus limiting the development of more
effective and explainable RNNs. To tackle the problem, in this paper, we
identify and analyze the internal and external factors that affect the memory
ability of RNNs, and propose a Semantic Euclidean Space to represent the
semantics expressed by a sequence. Based on the Semantic Euclidean Space, a
series of evaluation indicators are defined to measure the memory abilities of
different recurrent units and analyze their limitations. These evaluation
indicators also provide a useful guidance to select suitable sequence lengths
for different RNNs during training.
