Two Sparsities Are Better Than One: Unlocking the Performance Benefits
  of Sparse-Sparse Networks
  In principle, sparse neural networks should be significantly more efficient
than traditional dense networks. Neurons in the brain exhibit two types of
sparsity; they are sparsely interconnected and sparsely active. These two types
of sparsity, called weight sparsity and activation sparsity, when combined,
offer the potential to reduce the computational cost of neural networks by two
orders of magnitude. Despite this potential, today's neural networks deliver
only modest performance benefits using just weight sparsity, because
traditional computing hardware cannot efficiently process sparse networks. In
this article we introduce Complementary Sparsity, a novel technique that
significantly improves the performance of dual sparse networks on existing
hardware. We demonstrate that we can achieve high performance running
weight-sparse networks, and we can multiply those speedups by incorporating
activation sparsity. Using Complementary Sparsity, we show up to 100X
improvement in throughput and energy efficiency performing inference on FPGAs.
We analyze scalability and resource tradeoffs for a variety of kernels typical
of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our
results with Complementary Sparsity suggest that weight plus activation
sparsity can be a potent combination for efficiently scaling future AI models.
