The Power of Linear Recurrent Neural Networks
  Recurrent neural networks are a powerful means to cope with time series. We
show how linear, i.e., linearly activated recurrent neural networks (LRNNs) can
approximate any time-dependent function f(t) given by a number of function
values. The approximation can effectively be learned by simply solving a linear
equation system; no backpropagation or similar methods are needed. Furthermore,
the size of an LRNN can be reduced significantly in one step, after inspecting
the eigenvalues of the network transition matrix, by taking only the most
relevant components. Therefore, in contrast to others, we do not only learn
network weights but also the network architecture. LRNNs have interesting
properties: They end up in ellipse trajectories in the long run and allow the
prediction of further values and compact representations of functions. We
demonstrate this by several experiments, among them multiple superimposed
oscillators (MSO), robotic soccer, and predicting stock prices. LRNNs
outperform the previous state-of-the-art for the MSO task with a minimal number
of units.
