Low-rank Tensor Decomposition for Compression of Convolutional Neural
  Networks Using Funnel Regularization
  Tensor decomposition is one of the fundamental technique for model
compression of deep convolution neural networks owing to its ability to reveal
the latent relations among complex structures. However, most existing methods
compress the networks layer by layer, which cannot provide a satisfactory
solution to achieve global optimization. In this paper, we proposed a model
reduction method to compress the pre-trained networks using low-rank tensor
decomposition of the convolution layers. Our method is based on the
optimization techniques to select the proper ranks of decomposed network
layers. A new regularization method, called funnel function, is proposed to
suppress the unimportant factors during the compression, so the proper ranks
can be revealed much easier. The experimental results show that our algorithm
can reduce more model parameters than other tensor compression methods. For
ResNet18 with ImageNet2012, our reduced model can reach more than twi times
speed up in terms of GMAC with merely 0.7% Top-1 accuracy drop, which
outperforms most existing methods in both metrics.
