Differential Privacy Guarantees for Stochastic Gradient Langevin
  Dynamics
  We analyse the privacy leakage of noisy stochastic gradient descent by
modeling R\'enyi divergence dynamics with Langevin diffusions. Inspired by
recent work on non-stochastic algorithms, we derive similar desirable
properties in the stochastic setting. In particular, we prove that the privacy
loss converges exponentially fast for smooth and strongly convex objectives
under constant step size, which is a significant improvement over previous
DP-SGD analyses. We also extend our analysis to arbitrary sequences of varying
step sizes and derive new utility bounds. Last, we propose an implementation
and our experiments show the practical utility of our approach compared to
classical DP-SGD libraries.
