Low-Rank+Sparse Tensor Compression for Neural Networks
  Low-rank tensor compression has been proposed as a promising approach to
reduce the memory and compute requirements of neural networks for their
deployment on edge devices. Tensor compression reduces the number of parameters
required to represent a neural network weight by assuming network weights
possess a coarse higher-order structure. This coarse structure assumption has
been applied to compress large neural networks such as VGG and ResNet. However
modern state-of-the-art neural networks for computer vision tasks (i.e.
MobileNet, EfficientNet) already assume a coarse factorized structure through
depthwise separable convolutions, making pure tensor decomposition a less
attractive approach. We propose to combine low-rank tensor decomposition with
sparse pruning in order to take advantage of both coarse and fine structure for
compression. We compress weights in SOTA architectures (MobileNetv3,
EfficientNet, Vision Transformer) and compare this approach to sparse pruning
and tensor decomposition alone.
