An Overview of Neural Network Compression
  Overparameterized networks trained to convergence have shown impressive
performance in domains such as computer vision and natural language processing.
Pushing state of the art on salient tasks within these domains corresponds to
these models becoming larger and more difficult for machine learning
practitioners to use given the increasing memory and storage requirements, not
to mention the larger carbon footprint. Thus, in recent years there has been a
resurgence in model compression techniques, particularly for deep convolutional
neural networks and self-attention based networks such as the Transformer.
  Hence, this paper provides a timely overview of both old and current
compression techniques for deep neural networks, including pruning,
quantization, tensor decomposition, knowledge distillation and combinations
thereof.
  We assume a basic familiarity with deep learning architectures\footnote{For
an introduction to deep learning, see ~\citet{goodfellow2016deep}}, namely,
Recurrent Neural
Networks~\citep[(RNNs)][]{rumelhart1985learning,hochreiter1997long},
Convolutional Neural Networks~\citep{fukushima1980neocognitron}~\footnote{For
an up to date overview see~\citet{khan2019survey}} and Self-Attention based
networks~\citep{vaswani2017attention}\footnote{For a general overview of
self-attention networks, see ~\citet{chaudhari2019attentive}.},\footnote{For
more detail and their use in natural language processing,
see~\citet{hu2019introductory}}. Most of the papers discussed are proposed in
the context of at least one of these DNN architectures.
