On the Statistical Challenges of Echo State Networks and Some Potential
  Remedies
  Echo state networks are powerful recurrent neural networks. However, they are
often unstable and shaky, making the process of finding an good ESN for a
specific dataset quite hard. Obtaining a superb accuracy by using the Echo
State Network is a challenging task. We create, develop and implement a family
of predictably optimal robust and stable ensemble of Echo State Networks via
regularizing the training and perturbing the input. Furthermore, several
distributions of weights have been tried based on the shape to see if the shape
of the distribution has the impact for reducing the error. We found ESN can
track in short term for most dataset, but it collapses in the long run.
Short-term tracking with large size reservoir enables ESN to perform strikingly
with superior prediction. Based on this scenario, we go a further step to
aggregate many of ESNs into an ensemble to lower the variance and stabilize the
system by stochastic replications and bootstrapping of input data.
