Privacy Amplification via Iteration for Shuffled and Online PNSGD
  In this paper, we consider the framework of privacy amplification via
iteration, which is originally proposed by Feldman et al. and subsequently
simplified by Asoodeh et al. in their analysis via the contraction coefficient.
This line of work focuses on the study of the privacy guarantees obtained by
the projected noisy stochastic gradient descent (PNSGD) algorithm with hidden
intermediate updates. A limitation in the existing literature is that only the
early stopped PNSGD has been studied, while no result has been proved on the
more widely-used PNSGD applied on a shuffled dataset. Moreover, no scheme has
been yet proposed regarding how to decrease the injected noise when new data
are received in an online fashion. In this work, we first prove a privacy
guarantee for shuffled PNSGD, which is investigated asymptotically when the
noise is fixed for each sample size $n$ but reduced at a predetermined rate
when $n$ increases, in order to achieve the convergence of privacy loss. We
then analyze the online setting and provide a faster decaying scheme for the
magnitude of the injected noise that also guarantees the convergence of privacy
loss.
