Hybrid Backpropagation Parallel Reservoir Networks
  In many real-world applications, fully-differentiable RNNs such as LSTMs and
GRUs have been widely deployed to solve time series learning tasks. These
networks train via Backpropagation Through Time, which can work well in
practice but involves a biologically unrealistic unrolling of the network in
time for gradient updates, are computationally expensive, and can be hard to
tune. A second paradigm, Reservoir Computing, keeps the recurrent weight matrix
fixed and random. Here, we propose a novel hybrid network, which we call Hybrid
Backpropagation Parallel Echo State Network (HBP-ESN) which combines the
effectiveness of learning random temporal features of reservoirs with the
readout power of a deep neural network with batch normalization. We demonstrate
that our new network outperforms LSTMs and GRUs, including multi-layer "deep"
versions of these networks, on two complex real-world multi-dimensional time
series datasets: gesture recognition using skeleton keypoints from ChaLearn,
and the DEAP dataset for emotion recognition from EEG measurements. We show
also that the inclusion of a novel meta-ring structure, which we call HBP-ESN
M-Ring, achieves similar performance to one large reservoir while decreasing
the memory required by an order of magnitude. We thus offer this new hybrid
reservoir deep learning paradigm as a new alternative direction for RNN
learning of temporal or sequential data.
