An Empirical Study on Compressed Decentralized Stochastic Gradient
  Algorithms with Overparameterized Models
  This paper considers decentralized optimization with application to machine
learning on graphs. The growing size of neural network (NN) models has
motivated prior works on decentralized stochastic gradient algorithms to
incorporate communication compression. On the other hand, recent works have
demonstrated the favorable convergence and generalization properties of
overparameterized NNs. In this work, we present an empirical analysis on the
performance of compressed decentralized stochastic gradient (DSG) algorithms
with overparameterized NNs. Through simulations on an MPI network environment,
we observe that the convergence rates of popular compressed DSG algorithms are
robust to the size of NNs. Our findings suggest a gap between theories and
practice of the compressed DSG algorithms in the existing literature.
