R-FORCE: Robust Learning for Random Recurrent Neural Networks
  Random Recurrent Neural Networks (RRNN) are the simplest recurrent networks
to model and extract features from sequential data. The simplicity however
comes with a price; RRNN are known to be susceptible to diminishing/exploding
gradient problem when trained with gradient-descent based optimization. To
enhance robustness of RRNN, alternative training approaches have been proposed.
Specifically, FORCE learning approach proposed a recursive least squares
alternative to train RRNN and was shown to be applicable even for the
challenging task of target-learning, where the network is tasked with
generating dynamic patterns with no guiding input. While FORCE training
indicates that solving target-learning is possible, it appears to be effective
only in a specific regime of network dynamics (edge-of-chaos). We thereby
investigate whether initialization of RRNN connectivity according to a tailored
distribution can guarantee robust FORCE learning. We are able to generate such
distribution by inference of four generating principles constraining the
spectrum of the network Jacobian to remain in stability region. This
initialization along with FORCE learning provides a robust training method,
i.e., Robust-FORCE (R-FORCE). We validate R-FORCE performance on various target
functions for a wide range of network configurations and compare with
alternative methods. Our experiments indicate that R-FORCE facilitates
significantly more stable and accurate target-learning for a wide class of
RRNN. Such stability becomes critical in modeling multi-dimensional sequences
as we demonstrate on modeling time-series of human body joints during physical
movements.
