Hierarchical Architectures in Reservoir Computing Systems
  Reservoir computing (RC) offers efficient temporal data processing with a low
training cost by separating recurrent neural networks into a fixed network with
recurrent connections and a trainable linear network. The quality of the fixed
network, called reservoir, is the most important factor that determines the
performance of the RC system. In this paper, we investigate the influence of
the hierarchical reservoir structure on the properties of the reservoir and the
performance of the RC system. Analogous to deep neural networks, stacking
sub-reservoirs in series is an efficient way to enhance the nonlinearity of
data transformation to high-dimensional space and expand the diversity of
temporal information captured by the reservoir. These deep reservoir systems
offer better performance when compared to simply increasing the size of the
reservoir or the number of sub-reservoirs. Low frequency components are mainly
captured by the sub-reservoirs in later stage of the deep reservoir structure,
similar to observations that more abstract information can be extracted by
layers in the late stage of deep neural networks. When the total size of the
reservoir is fixed, tradeoff between the number of sub-reservoirs and the size
of each sub-reservoir needs to be carefully considered, due to the degraded
ability of individual sub-reservoirs at small sizes. Improved performance of
the deep reservoir structure alleviates the difficulty of implementing the RC
system on hardware systems.
