Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
  Large deep neural network (DNN) models pose the key challenge to energy
efficiency due to the significantly higher energy consumption of off-chip DRAM
accesses than arithmetic or SRAM operations. It motivates the intensive
research on model compression with two main approaches. Weight pruning
leverages the redundancy in the number of weights and can be performed in a
non-structured, which has higher flexibility and pruning rate but incurs index
accesses due to irregular weights, or structured manner, which preserves the
full matrix structure with lower pruning rate. Weight quantization leverages
the redundancy in the number of bits in weights. Compared to pruning,
quantization is much more hardware-friendly, and has become a "must-do" step
for FPGA and ASIC implementations. This paper provides a definitive answer to
the question for the first time. First, we build ADMM-NN-S by extending and
enhancing ADMM-NN, a recently proposed joint weight pruning and quantization
framework. Second, we develop a methodology for fair and fundamental comparison
of non-structured and structured pruning in terms of both storage and
computation efficiency. Our results show that ADMM-NN-S consistently
outperforms the prior art: (i) it achieves 348x, 36x, and 8x overall weight
pruning on LeNet-5, AlexNet, and ResNet-50, respectively, with (almost) zero
accuracy loss; (ii) we demonstrate the first fully binarized (for all layers)
DNNs can be lossless in accuracy in many cases. These results provide a strong
baseline and credibility of our study. Based on the proposed comparison
framework, with the same accuracy and quantization, the results show that
non-structrued pruning is not competitive in terms of both storage and
computation efficiency. Thus, we conclude that non-structured pruning is
considered harmful. We urge the community not to continue the DNN inference
acceleration for non-structured sparsity.
