Sparsity in Reservoir Computing Neural Networks
  Reservoir Computing (RC) is a well-known strategy for designing Recurrent
Neural Networks featured by striking efficiency of training. The crucial aspect
of RC is to properly instantiate the hidden recurrent layer that serves as
dynamical memory to the system. In this respect, the common recipe is to create
a pool of randomly and sparsely connected recurrent neurons. While the aspect
of sparsity in the design of RC systems has been debated in the literature, it
is nowadays understood mainly as a way to enhance the efficiency of
computation, exploiting sparse matrix operations. In this paper, we empirically
investigate the role of sparsity in RC network design under the perspective of
the richness of the developed temporal representations. We analyze both
sparsity in the recurrent connections, and in the connections from the input to
the reservoir. Our results point out that sparsity, in particular in
input-reservoir connections, has a major role in developing internal temporal
representations that have a longer short-term memory of past inputs and a
higher dimension.
