Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time
  Mobile Acceleration
  Weight pruning is an effective model compression technique to tackle the
challenges of achieving real-time deep neural network (DNN) inference on mobile
devices. However, prior pruning schemes have limited application scenarios due
to accuracy degradation, difficulty in leveraging hardware acceleration, and/or
restriction on certain types of DNN layers. In this paper, we propose a
general, fine-grained structured pruning scheme and corresponding compiler
optimizations that are applicable to any type of DNN layer while achieving high
accuracy and hardware inference performance. With the flexibility of applying
different pruning schemes to different layers enabled by our compiler
optimizations, we further probe into the new problem of determining the
best-suited pruning scheme considering the different acceleration and accuracy
performance of various pruning schemes. Two pruning scheme mapping methods, one
is search-based and the other is rule-based, are proposed to automatically
derive the best-suited pruning regularity and block size for each layer of any
given DNN. Experimental results demonstrate that our pruning scheme mapping
methods, together with the general fine-grained structured pruning scheme,
outperform the state-of-the-art DNN optimization framework with up to
2.48$\times$ and 1.73$\times$ DNN inference acceleration on CIFAR-10 and
ImageNet dataset without accuracy loss.
