Dual-side Sparse Tensor Core
  Leveraging sparsity in deep neural network (DNN) models is promising for
accelerating model inference. Yet existing GPUs can only leverage the sparsity
from weights but not activations, which are dynamic, unpredictable, and hence
challenging to exploit. In this work, we propose a novel architecture to
efficiently harness the dual-side sparsity (i.e., weight and activation
sparsity). We take a systematic approach to understand the (dis)advantages of
previous sparsity-related architectures and propose a novel, unexplored
paradigm that combines outer-product computation primitive and bitmap-based
encoding format. We demonstrate the feasibility of our design with minimal
changes to the existing production-scale inner-product-based Tensor Core. We
propose a set of novel ISA extensions and co-design the matrix-matrix
multiplication and convolution algorithms, which are the two dominant
computation patterns in today's DNN models, to exploit our new dual-side sparse
Tensor Core. Our evaluation shows that our design can fully unleash the
dual-side DNN sparsity and improve the performance by up to one order of
magnitude with \hl{small} hardware overhead.
