Ps and Qs: Quantization-aware pruning for efficient low latency neural
  network inference
  Efficient machine learning implementations optimized for inference in
hardware have wide-ranging benefits, depending on the application, from lower
inference latency to higher data throughput and reduced energy consumption. Two
popular techniques for reducing computation in neural networks are pruning,
removing insignificant synapses, and quantization, reducing the precision of
the calculations. In this work, we explore the interplay between pruning and
quantization during the training of neural networks for ultra low latency
applications targeting high energy physics use cases. Techniques developed for
this study have potential applications across many other domains. We study
various configurations of pruning during quantization-aware training, which we
term quantization-aware pruning, and the effect of techniques like
regularization, batch normalization, and different pruning schemes on
performance, computational complexity, and information content metrics. We find
that quantization-aware pruning yields more computationally efficient models
than either pruning or quantization alone for our task. Further,
quantization-aware pruning typically performs similar to or better in terms of
computational efficiency compared to other neural architecture search
techniques like Bayesian optimization. Surprisingly, while networks with
different training configurations can have similar performance for the
benchmark application, the information content in the network can vary
significantly, affecting its generalizability.
