Universality and approximation bounds for echo state networks with
  random weights
  We study the uniform approximation of echo state networks with randomly
generated internal weights. These models, in which only the readout weights are
optimized during training, have made empirical success in learning dynamical
systems. We address the representational capacity of these models by showing
that they are universal under weak conditions. Our main result gives a
sufficient condition for the activation function and a sampling procedure for
the internal weights so that echo state networks can approximate any continuous
casual time-invariant operators with high probability. In particular, for ReLU
activation, we quantify the approximation error of echo state networks for
sufficiently regular operators.
