Bayesian optimization of hyper-parameters in reservoir computing
  We describe a method for searching the optimal hyper-parameters in reservoir
computing, which consists of a Gaussian process with Bayesian optimization. It
provides an alternative to other frequently used optimization methods such as
grid, random, or manual search. In addition to a set of optimal
hyper-parameters, the method also provides a probability distribution of the
cost function as a function of the hyper-parameters. We apply this method to
two types of reservoirs: nonlinear delay nodes and echo state networks. It
shows excellent performance on all considered benchmarks, either matching or
significantly surpassing results found in the literature. In general, the
algorithm achieves optimal results in fewer iterations when compared to other
optimization methods. We have optimized up to six hyper-parameters
simultaneously, which would have been infeasible using, e.g., grid search. Due
to its automated nature, this method significantly reduces the need for expert
knowledge when optimizing the hyper-parameters in reservoir computing. Existing
software libraries for Bayesian optimization, such as Spearmint, make the
implementation of the algorithm straightforward. A fork of the Spearmint
framework along with a tutorial on how to use it in practice is available at
https://bitbucket.org/uhasseltmachinelearning/spearmint/
