Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box
  Floating-Point Transformer Models
  Increasingly larger and better Transformer models keep advancing
state-of-the-art accuracy and capability for Natural Language Processing
applications. These models demand more computational power, storage, and
energy. Mokey reduces the footprint of state-of-the-art 32-bit or 16-bit
floating-point transformer models by quantizing all values to 4-bit indexes
into dictionaries of representative 16-bit fixed-point centroids. Mokey does
not need fine-tuning, an essential feature as often the training resources or
datasets are not available to many. Exploiting the range of values that
naturally occur in transformer models, Mokey selects centroid values to also
fit an exponential curve. This unique feature enables Mokey to replace the bulk
of the original multiply-accumulate operations with narrow 3b fixed-point
additions resulting in an area- and energy-efficient hardware accelerator
design. Over a set of state-of-the-art transformer models, the Mokey
accelerator delivers an order of magnitude improvements in energy efficiency
over a Tensor Cores-based accelerator while improving performance by at least
$4\times$ and as much as $15\times$ depending on the model and on-chip
buffering capacity. Optionally, Mokey can be used as a memory compression
assist for any other accelerator, transparently stashing wide floating-point or
fixed-point activations or weights into narrow 4-bit indexes. Mokey proves
superior to prior state-of-the-art quantization methods for Transformers.
