Unsupervised Reservoir Computing for Solving Ordinary Differential
  Equations
  There is a wave of interest in using unsupervised neural networks for solving
differential equations. The existing methods are based on feed-forward
networks, {while} recurrent neural network differential equation solvers have
not yet been reported. We introduce an unsupervised reservoir computing (RC),
an echo-state recurrent neural network capable of discovering approximate
solutions that satisfy ordinary differential equations (ODEs). We suggest an
approach to calculate time derivatives of recurrent neural network outputs
without using backpropagation. The internal weights of an RC are fixed, while
only a linear output layer is trained, yielding efficient training. However, RC
performance strongly depends on finding the optimal hyper-parameters, which is
a computationally expensive process. We use Bayesian optimization to
efficiently discover optimal sets in a high-dimensional hyper-parameter space
and numerically show that one set is robust and can be used to solve an ODE for
different initial conditions and time ranges. A closed-form formula for the
optimal output weights is derived to solve first order linear equations in a
backpropagation-free learning process. We extend the RC approach by solving
nonlinear system of ODEs using a hybrid optimization method consisting of
gradient descent and Bayesian optimization. Evaluation of linear and nonlinear
systems of equations demonstrates the efficiency of the RC ODE solver.
