Connecting Weighted Automata, Tensor Networks and Recurrent Neural
  Networks through Spectral Learning
  In this paper, we present connections between three models used in different
research fields: weighted finite automata~(WFA) from formal languages and
linguistics, recurrent neural networks used in machine learning, and tensor
networks which encompasses a set of optimization techniques for high-order
tensors used in quantum physics and numerical analysis. We first present an
intrinsic relation between WFA and the tensor train decomposition, a particular
form of tensor network. This relation allows us to exhibit a novel low rank
structure of the Hankel matrix of a function computed by a WFA and to design an
efficient spectral learning algorithm leveraging this structure to scale the
algorithm up to very large Hankel matrices.We then unravel a fundamental
connection between WFA and second-orderrecurrent neural networks~(2-RNN): in
the case of sequences of discrete symbols, WFA and 2-RNN with linear
activationfunctions are expressively equivalent. Leveraging this equivalence
result combined with the classical spectral learning algorithm for weighted
automata, we introduce the first provable learning algorithm for linear 2-RNN
defined over sequences of continuous input vectors.This algorithm relies on
estimating low rank sub-blocks of the Hankel tensor, from which the parameters
of a linear 2-RNN can be provably recovered. The performances of the proposed
learning algorithm are assessed in a simulation study on both synthetic and
real-world data.
