Cascaded Projection: End-to-End Network Compression and Acceleration
  We propose a data-driven approach for deep convolutional neural network
compression that achieves high accuracy with high throughput and low memory
requirements. Current network compression methods either find a low-rank
factorization of the features that requires more memory, or select only a
subset of features by pruning entire filter channels. We propose the Cascaded
Projection (CaP) compression method that projects the output and input filter
channels of successive layers to a unified low dimensional space based on a
low-rank projection. We optimize the projection to minimize classification loss
and the difference between the next layer's features in the compressed and
uncompressed networks. To solve this non-convex optimization problem we propose
a new optimization method of a proxy matrix using backpropagation and
Stochastic Gradient Descent (SGD) with geometric constraints. Our cascaded
projection approach leads to improvements in all critical areas of network
compression: high accuracy, low memory consumption, low parameter count and
high processing speed. The proposed CaP method demonstrates state-of-the-art
results compressing VGG16 and ResNet networks with over 4x reduction in the
number of computations and excellent performance in top-5 accuracy on the
ImageNet dataset before and after fine-tuning.
