Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient
  Descent
  What is the information leakage of an iterative randomized learning algorithm
about its training data, when the internal state of the algorithm is
\emph{private}? How much is the contribution of each specific training epoch to
the information leakage through the released model? We study this problem for
noisy gradient descent algorithms, and model the \emph{dynamics} of R\'enyi
differential privacy loss throughout the training process. Our analysis traces
a provably \emph{tight} bound on the R\'enyi divergence between the pair of
probability distributions over parameters of models trained on neighboring
datasets. We prove that the privacy loss converges exponentially fast, for
smooth and strongly convex loss functions, which is a significant improvement
over composition theorems (which over-estimate the privacy loss by
upper-bounding its total value over all intermediate gradient computations).
For Lipschitz, smooth, and strongly convex loss functions, we prove optimal
utility with a small gradient complexity for noisy gradient descent algorithms.
