Richness of Deep Echo State Network Dynamics
  Reservoir Computing (RC) is a popular methodology for the efficient design of
Recurrent Neural Networks (RNNs). Recently, the advantages of the RC approach
have been extended to the context of multi-layered RNNs, with the introduction
of the Deep Echo State Network (DeepESN) model. In this paper, we study the
quality of state dynamics in progressively higher layers of DeepESNs, using
tools from the areas of information theory and numerical analysis. Our
experimental results on RC benchmark datasets reveal the fundamental role
played by the strength of inter-reservoir connections to increasingly enrich
the representations developed in higher layers. Our analysis also gives
interesting insights into the possibility of effective exploitation of training
algorithms based on stochastic gradient descent in the RC field.
