Can Stochastic Gradient Langevin Dynamics Provide Differential Privacy
  for Deep Learning?
  Bayesian learning via Stochastic Gradient Langevin Dynamics (SGLD) has been
suggested for differentially private learning. While previous research provides
differential privacy bounds for SGLD at the initial steps of the algorithm or
when close to convergence, the question of what differential privacy guarantees
can be made in between remains unanswered. This interim region is of great
importance, especially for Bayesian neural networks, as it is hard to guarantee
convergence to the posterior. This paper shows that using SGLD might result in
unbounded privacy loss for this interim region, even when sampling from the
posterior is as differentially private as desired.
