Exploring Bit-Slice Sparsity in Deep Neural Networks for Efficient
  ReRAM-Based Deployment
  Emerging resistive random-access memory (ReRAM) has recently been intensively
investigated to accelerate the processing of deep neural networks (DNNs). Due
to the in-situ computation capability, analog ReRAM crossbars yield significant
throughput improvement and energy reduction compared to traditional digital
methods. However, the power hungry analog-to-digital converters (ADCs) prevent
the practical deployment of ReRAM-based DNN accelerators on end devices with
limited chip area and power budget. We observe that due to the limited
bit-density of ReRAM cells, DNN weights are bit sliced and correspondingly
stored on multiple ReRAM bitlines. The accumulated current on bitlines resulted
by weights directly dictates the overhead of ADCs. As such, bitwise weight
sparsity rather than the sparsity of the full weight, is desirable for
efficient ReRAM deployment. In this work, we propose bit-slice L1, the first
algorithm to induce bit-slice sparsity during the training of dynamic
fixed-point DNNs. Experiment results show that our approach achieves 2x
sparsity improvement compared to previous algorithms. The resulting sparsity
allows the ADC resolution to be reduced to 1-bit of the most significant
bit-slice and down to 3-bit for the others bits, which significantly speeds up
processing and reduces power and area overhead.
