Tensor train decompositions on recurrent networks
  Recurrent neural networks (RNN) such as long-short-term memory (LSTM)
networks are essential in a multitude of daily live tasks such as speech,
language, video, and multimodal learning. The shift from cloud to edge
computation intensifies the need to contain the growth of RNN parameters.
Current research on RNN shows that despite the performance obtained on
convolutional neural networks (CNN), keeping a good performance in compressed
RNNs is still a challenge. Most of the literature on compression focuses on
CNNs using matrix product (MPO) operator tensor trains. However, matrix product
state (MPS) tensor trains have more attractive features than MPOs, in terms of
storage reduction and computing time at inference. We show that MPS tensor
trains should be at the forefront of LSTM network compression through a
theoretical analysis and practical experiments on NLP task.
