On the Influence of Bias-Correction on Distributed Stochastic
  Optimization
  Various bias-correction methods such as EXTRA, gradient tracking methods, and
exact diffusion have been proposed recently to solve distributed {\em
deterministic} optimization problems. These methods employ constant step-sizes
and converge linearly to the {\em exact} solution under proper conditions.
However, their performance under stochastic and adaptive settings is less
explored. It is still unknown {\em whether}, {\em when} and {\em why} these
bias-correction methods can outperform their traditional counterparts (such as
consensus and diffusion) with noisy gradient and constant step-sizes.
  This work studies the performance of exact diffusion under the stochastic and
adaptive setting, and provides conditions under which exact diffusion has
superior steady-state mean-square deviation (MSD) performance than traditional
algorithms without bias-correction. In particular, it is proven that this
superiority is more evident over sparsely-connected network topologies such as
lines, cycles, or grids. Conditions are also provided under which exact
diffusion method match or may even degrade the performance of traditional
methods. Simulations are provided to validate the theoretical findings.
