DNN gradient lossless compression: Can GenNorm be the answer?
  In this paper, the problem of optimal gradient lossless compression in Deep
Neural Network (DNN) training is considered. Gradient compression is relevant
in many distributed DNN training scenarios, including the recently popular
federated learning (FL) scenario in which each remote users are connected to
the parameter server (PS) through a noiseless but rate limited channel. In
distributed DNN training, if the underlying gradient distribution is available,
classical lossless compression approaches can be used to reduce the number of
bits required for communicating the gradient entries. Mean field analysis has
suggested that gradient updates can be considered as independent random
variables, while Laplace approximation can be used to argue that gradient has a
distribution approximating the normal (Norm) distribution in some regimes. In
this paper we argue that, for some networks of practical interest, the gradient
entries can be well modelled as having a generalized normal (GenNorm)
distribution. We provide numerical evaluations to validate that the hypothesis
GenNorm modelling provides a more accurate prediction of the DNN gradient tail
distribution. Additionally, this modeling choice provides concrete improvement
in terms of lossless compression of the gradients when applying classical
fix-to-variable lossless coding algorithms, such as Huffman coding, to the
quantized gradient updates. This latter results indeed provides an effective
compression strategy with low memory and computational complexity that has
great practical relevance in distributed DNN training scenarios.
