Robust Optimization and Validation of Echo State Networks for learning
  chaotic dynamics
  An approach to the time-accurate prediction of chaotic solutions is by
learning temporal patterns from data. Echo State Networks (ESNs), which are a
class of Reservoir Computing, can accurately predict the chaotic dynamics well
beyond the predictability time. Existing studies, however, also showed that
small changes in the hyperparameters may markedly affect the network's
performance. The aim of this paper is to assess and improve the robustness of
Echo State Networks for the time-accurate prediction of chaotic solutions. The
goal is three-fold. First, we investigate the robustness of routinely used
validation strategies. Second, we propose the Recycle Validation, and the
chaotic versions of existing validation strategies, to specifically tackle the
forecasting of chaotic systems. Third, we compare Bayesian optimization with
the traditional Grid Search for optimal hyperparameter selection. Numerical
tests are performed on two prototypical nonlinear systems that have both
chaotic and quasiperiodic solutions. Both model-free and model-informed Echo
State Networks are analysed. By comparing the network's robustness in learning
chaotic versus quasiperiodic solutions, we highlight fundamental challenges in
learning chaotic solutions. The proposed validation strategies, which are based
on the dynamical systems properties of chaotic time series, are shown to
outperform the state-of-the-art validation strategies. Because the strategies
are principled-they are based on chaos theory such as the Lyapunov time-they
can be applied to other Recurrent Neural Networks architectures with little
modification. This work opens up new possibilities for the robust design and
application of Echo State Networks, and Recurrent Neural Networks, to the
time-accurate prediction of chaotic systems.
