Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs
  Deep Neural Networks (DNNs) are widely used in many applications domains.
However, they require a vast amount of computations and memory accesses to
deliver outstanding accuracy. In this paper, we propose a scheme to predict
whether the output of each ReLu activated neuron will be a zero or a positive
number in order to skip the computation of those neurons that will likely
output a zero. Our predictor, named Mixture-of-Rookies, combines two
inexpensive components. The first one exploits the high linear correlation
between binarized (1-bit) and full-precision (8-bit) dot products, whereas the
second component clusters together neurons that tend to output zero at the same
time. We propose a novel clustering scheme based on the analysis of angles, as
the sign of the dot product of two vectors depends on the cosine of the angle
between them. We implement our hybrid zero output predictor on top of a
state-of-the-art DNN accelerator. Experimental results show that our scheme
introduces a small area overhead of 5.3% while achieving a speedup of 1.2x and
reducing energy consumption by 16.5% on average for a set of diverse DNNs.
