Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive
  ADMM
  Weight quantization is one of the most important techniques of Deep Neural
Networks (DNNs) model compression method. A recent work using systematic
framework of DNN weight quantization with the advanced optimization algorithm
ADMM (Alternating Direction Methods of Multipliers) achieves one of
state-of-art results in weight quantization. In this work, we first extend such
ADMM-based framework to guarantee solution feasibility and we have further
developed a multi-step, progressive DNN weight quantization framework, with
dual benefits of (i) achieving further weight quantization thanks to the
special property of ADMM regularization, and (ii) reducing the search space
within each step. Extensive experimental results demonstrate the superior
performance compared with prior work. Some highlights: we derive the first
lossless and fully binarized (for all layers) LeNet-5 for MNIST; And we derive
the first fully binarized (for all layers) VGG-16 for CIFAR-10 and ResNet for
ImageNet with reasonable accuracy loss.
