Automatic Neural Network Compression by Sparsity-Quantization Joint
  Learning: A Constrained Optimization-based Approach
  Deep Neural Networks (DNNs) are applied in a wide range of usecases. There is
an increased demand for deploying DNNs on devices that do not have abundant
resources such as memory and computation units. Recently, network compression
through a variety of techniques such as pruning and quantization have been
proposed to reduce the resource requirement. A key parameter that all existing
compression techniques are sensitive to is the compression ratio (e.g., pruning
sparsity, quantization bitwidth) of each layer. Traditional solutions treat the
compression ratios of each layer as hyper-parameters, and tune them using human
heuristic. Recent researchers start using black-box hyper-parameter
optimizations, but they will introduce new hyper-parameters and have efficiency
issue. In this paper, we propose a framework to jointly prune and quantize the
DNNs automatically according to a target model size without using any
hyper-parameters to manually set the compression ratio for each layer. In the
experiments, we show that our framework can compress the weights data of
ResNet-50 to be 836$\times$ smaller without accuracy loss on CIFAR-10, and
compress AlexNet to be 205$\times$ smaller without accuracy loss on ImageNet
classification.
