A Systematic DNN Weight Pruning Framework using Alternating Direction
  Method of Multipliers
  Weight pruning methods for deep neural networks (DNNs) have been investigated
recently, but prior work in this area is mainly heuristic, iterative pruning,
thereby lacking guarantees on the weight reduction ratio and convergence time.
To mitigate these limitations, we present a systematic weight pruning framework
of DNNs using the alternating direction method of multipliers (ADMM). We first
formulate the weight pruning problem of DNNs as a nonconvex optimization
problem with combinatorial constraints specifying the sparsity requirements,
and then adopt the ADMM framework for systematic weight pruning. By using ADMM,
the original nonconvex optimization problem is decomposed into two subproblems
that are solved iteratively. One of these subproblems can be solved using
stochastic gradient descent, the other can be solved analytically. Besides, our
method achieves a fast convergence rate.
  The weight pruning results are very promising and consistently outperform the
prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times
weight reduction without accuracy loss. On the AlexNet model for the ImageNet
data set, we achieve 21 times weight reduction without accuracy loss. When we
focus on the convolutional layer pruning for computation reductions, we can
reduce the total computation by five times compared with the prior work
(achieving a total of 13.4 times weight reduction in convolutional layers). Our
models and codes are released at https://github.com/KaiqiZhang/admm-pruning
