Determination of the edge of criticality in echo state networks through
  Fisher information maximization
  It is a widely accepted fact that the computational capability of recurrent
neural networks is maximized on the so-called "edge of criticality". Once the
network operates in this configuration, it performs efficiently on a specific
application both in terms of (i) low prediction error and (ii) high short-term
memory capacity. Since the behavior of recurrent networks is strongly
influenced by the particular input signal driving the dynamics, a universal,
application-independent method for determining the edge of criticality is still
missing. In this paper, we aim at addressing this issue by proposing a
theoretically motivated, unsupervised method based on Fisher information for
determining the edge of criticality in recurrent neural networks. It is proven
that Fisher information is maximized for (finite-size) systems operating in
such critical regions. However, Fisher information is notoriously difficult to
compute and either requires the probability density function or the conditional
dependence of the system states with respect to the model parameters. The paper
takes advantage of a recently-developed non-parametric estimator of the Fisher
information matrix and provides a method to determine the critical region of
echo state networks, a particular class of recurrent networks. The considered
control parameters, which indirectly affect the echo state network performance,
are explored to identify those configurations lying on the edge of criticality
and, as such, maximizing Fisher information and computational performance.
Experimental results on benchmarks and real-world data demonstrate the
effectiveness of the proposed method.
