Generalized Tensor Models for Recurrent Neural Networks
  Recurrent Neural Networks (RNNs) are very successful at solving challenging
problems with sequential data. However, this observed efficiency is not yet
entirely explained by theory. It is known that a certain class of
multiplicative RNNs enjoys the property of depth efficiency --- a shallow
network of exponentially large width is necessary to realize the same score
function as computed by such an RNN. Such networks, however, are not very often
applied to real life tasks. In this work, we attempt to reduce the gap between
theory and practice by extending the theoretical analysis to RNNs which employ
various nonlinearities, such as Rectified Linear Unit (ReLU), and show that
they also benefit from properties of universality and depth efficiency. Our
theoretical results are verified by a series of extensive computational
experiments.
