A$^3$: Accelerating Attention Mechanisms in Neural Networks with
  Approximation
  With the increasing computational demands of neural networks, many hardware
accelerators for the neural networks have been proposed. Such existing neural
network accelerators often focus on popular neural network types such as
convolutional neural networks (CNNs) and recurrent neural networks (RNNs);
however, not much attention has been paid to attention mechanisms, an emerging
neural network primitive that enables neural networks to retrieve most relevant
information from a knowledge-base, external memory, or past states. The
attention mechanism is widely adopted by many state-of-the-art neural networks
for computer vision, natural language processing, and machine translation, and
accounts for a large portion of total execution time. We observe today's
practice of implementing this mechanism using matrix-vector multiplication is
suboptimal as the attention mechanism is semantically a content-based search
where a large portion of computations ends up not being used. Based on this
observation, we design and architect A3, which accelerates attention mechanisms
in neural networks with algorithmic approximation and hardware specialization.
Our proposed accelerator achieves multiple orders of magnitude improvement in
energy efficiency (performance/watt) as well as substantial speedup over the
state-of-the-art conventional hardware.
