Analysis of Memory Capacity for Deep Echo State Networks
  In this paper, the echo state network (ESN) memory capacity, which represents
the amount of input data an ESN can store, is analyzed for a new type of deep
ESNs. In particular, two deep ESN architectures are studied. First, a parallel
deep ESN is proposed in which multiple reservoirs are connected in parallel
allowing them to average outputs of multiple ESNs, thus decreasing the
prediction error. Then, a series architecture ESN is proposed in which ESN
reservoirs are placed in cascade that the output of each ESN is the input of
the next ESN in the series. This series ESN architecture can capture more
features between the input sequence and the output sequence thus improving the
overall prediction accuracy. Fundamental analysis shows that the memory
capacity of parallel ESNs is equivalent to that of a traditional shallow ESN,
while the memory capacity of series ESNs is smaller than that of a traditional
shallow ESN.In terms of normalized root mean square error, simulation results
show that the parallel deep ESN achieves 38.5% reduction compared to the
traditional shallow ESN while the series deep ESN achieves 16.8% reduction.
