Low-Rank Embedding of Kernels in Convolutional Neural Networks under
  Random Shuffling
  Although the convolutional neural networks (CNNs) have become popular for
various image processing and computer vision task recently, it remains a
challenging problem to reduce the storage cost of the parameters for
resource-limited platforms. In the previous studies, tensor decomposition (TD)
has achieved promising compression performance by embedding the kernel of a
convolutional layer into a low-rank subspace. However the employment of TD is
naively on the kernel or its specified variants. Unlike the conventional
approaches, this paper shows that the kernel can be embedded into more general
or even random low-rank subspaces. We demonstrate this by compressing the
convolutional layers via randomly-shuffled tensor decomposition (RsTD) for a
standard classification task using CIFAR-10. In addition, we analyze how the
spatial similarity of the training data influences the low-rank structure of
the kernels. The experimental results show that the CNN can be significantly
compressed even if the kernels are randomly shuffled. Furthermore, the
RsTD-based method yields more stable classification accuracy than the
conventional TD-based methods in a large range of compression ratios.
