IntSGD: Adaptive Floatless Compression of Stochastic Gradients
  We propose a family of adaptive integer compression operators for distributed
Stochastic Gradient Descent (SGD) that do not communicate a single float. This
is achieved by multiplying floating-point vectors with a number known to every
device and then rounding to integers. In contrast to the prior work on integer
compression for SwitchML by Sapio et al. (2021), our IntSGD method is provably
convergent and computationally cheaper as it estimates the scaling of vectors
adaptively. Our theory shows that the iteration complexity of IntSGD matches
that of SGD up to constant factors for both convex and non-convex, smooth and
non-smooth functions, with and without overparameterization. Moreover, our
algorithm can also be tailored for the popular all-reduce primitive and shows
promising empirical performance.
