Enabling Retrain-free Deep Neural Network Pruning using Surrogate
  Lagrangian Relaxation
  Network pruning is a widely used technique to reduce computation cost and
model size for deep neural networks. However, the typical three-stage pipeline,
i.e., training, pruning and retraining (fine-tuning) significantly increases
the overall training trails. In this paper, we develop a systematic
weight-pruning optimization approach based on Surrogate Lagrangian relaxation
(SLR), which is tailored to overcome difficulties caused by the discrete nature
of the weight-pruning problem while ensuring fast convergence. We further
accelerate the convergence of the SLR by using quadratic penalties. Model
parameters obtained by SLR during the training phase are much closer to their
optimal values as compared to those obtained by other state-of-the-art methods.
We evaluate the proposed method on image classification tasks, i.e., ResNet-18
and ResNet-50 using ImageNet, and ResNet-18, ResNet-50 and VGG-16 using
CIFAR-10, as well as object detection tasks, i.e., YOLOv3 and YOLOv3-tiny using
COCO 2014 and Ultra-Fast-Lane-Detection using TuSimple lane detection dataset.
Experimental results demonstrate that our SLR-based weight-pruning optimization
approach achieves higher compression rate than state-of-the-arts under the same
accuracy requirement. It also achieves a high model accuracy even at the
hard-pruning stage without retraining (reduces the traditional three-stage
pruning to two-stage). Given a limited budget of retraining epochs, our
approach quickly recovers the model accuracy.
