Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?
  In federated learning (FL), a global model is trained at a Parameter Server
(PS) by aggregating model updates obtained from multiple remote learners.
Generally, the communication between the remote users and the PS is
rate-limited, while the transmission from the PS to the remote users are
unconstrained. The FL setting gives rise to the distributed learning scenario
in which the updates from the remote learners have to be compressed so as to
meet communication rate constraints in the uplink transmission toward the PS.
For this problem, one wishes to compress the model updates so as to minimize
the loss in accuracy resulting from the compression error. In this paper, we
take a rate-distortion approach to address the compressor design problem for
the distributed training of deep neural networks (DNNs). In particular, we
define a measure of the compression performance under communication-rate
constraints -- the \emph{per-bit accuracy} -- which addresses the ultimate
improvement of accuracy that a bit of communication brings to the centralized
model. In order to maximize the per-bit accuracy, we consider modeling the DNN
gradient updates at remote learners as a generalized normal distribution. Under
this assumption on the DNN gradient distribution, we propose a class of
distortion measures to aid the design of quantizers for the compression of the
model updates. We argue that this family of distortion measures, which we refer
to as "$M$-magnitude weighted $L_2$" norm, captures the practitioner's
intuition in the choice of gradient compressor. Numerical simulations are
provided to validate the proposed approach for the CIFAR-10 dataset.
