Federated Stochastic Gradient Langevin Dynamics
  Stochastic gradient MCMC methods, such as stochastic gradient Langevin
dynamics (SGLD), employ fast but noisy gradient estimates to enable large-scale
posterior sampling. Although we can easily extend SGLD to distributed settings,
it suffers from two issues when applied to federated non-IID data. First, the
variance of these estimates increases significantly. Second, delaying
communication causes the Markov chains to diverge from the true posterior even
for very simple models. To alleviate both these problems, we propose conducive
gradients, a simple mechanism that combines local likelihood approximations to
correct gradient updates. Notably, conducive gradients are easy to compute, and
since we only calculate the approximations once, they incur negligible
overhead. We apply conducive gradients to distributed stochastic gradient
Langevin dynamics (DSGLD) and call the resulting method federated stochastic
gradient Langevin dynamics (FSGLD). We demonstrate that our approach can handle
delayed communication rounds, converging to the target posterior in cases where
DSGLD fails. We also show that FSGLD outperforms DSGLD for non-IID federated
data with experiments on metric learning and neural networks.
