Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor
  Decomposition
  Recurrent Neural Networks (RNNs) have been widely used in sequence analysis
and modeling. However, when processing high-dimensional data, RNNs typically
require very large model sizes, thereby bringing a series of deployment
challenges. Although the state-of-the-art tensor decomposition approaches can
provide good model compression performance, these existing methods are still
suffering some inherent limitations, such as restricted representation
capability and insufficient model complexity reduction. To overcome these
limitations, in this paper we propose to develop compact RNN models using
Hierarchical Tucker (HT) decomposition. HT decomposition brings strong
hierarchical structure to the decomposed RNN models, which is very useful and
important for enhancing the representation capability. Meanwhile, HT
decomposition provides higher storage and computational cost reduction than the
existing tensor decomposition approaches for RNN compression. Our experimental
results show that, compared with the state-of-the-art compressed RNN models,
such as TT-LSTM, TR-LSTM and BT-LSTM, our proposed HT-based LSTM (HT-LSTM),
consistently achieves simultaneous and significant increases in both
compression ratio and test accuracy on different datasets.
