On Communication Compression for Distributed Optimization on
  Heterogeneous Data
  Lossy gradient compression, with either unbiased or biased compressors, has
become a key tool to avoid the communication bottleneck in centrally
coordinated distributed training of machine learning models. We analyze the
performance of two standard and general types of methods: (i) distributed
quantized SGD (D-QSGD) with arbitrary unbiased quantizers and (ii) distributed
SGD with error-feedback and biased compressors (D-EF-SGD) in the heterogeneous
(non-iid) data setting. Our results indicate that D-EF-SGD is much less
affected than D-QSGD by non-iid data, but both methods can suffer a slowdown if
data-skewness is high. We further study two alternatives that are not (or much
less) affected by heterogenous data distributions: first, a recently proposed
method that is effective on strongly convex problems, and secondly, we point
out a more general approach that is applicable to linear compressors only but
effective in all considered scenarios.
