Concentrated Differentially Private Gradient Descent with Adaptive
  per-Iteration Privacy Budget
  Iterative algorithms, like gradient descent, are common tools for solving a
variety of problems, such as model fitting. For this reason, there is interest
in creating differentially private versions of them. However, their conversion
to differentially private algorithms is often naive. For instance, a fixed
number of iterations are chosen, the privacy budget is split evenly among them,
and at each iteration, parameters are updated with a noisy gradient. In this
paper, we show that gradient-based algorithms can be improved by a more careful
allocation of privacy budget per iteration. Intuitively, at the beginning of
the optimization, gradients are expected to be large, so that they do not need
to be measured as accurately. However, as the parameters approach their optimal
values, the gradients decrease and hence need to be measured more accurately.
We add a basic line-search capability that helps the algorithm decide when more
accurate gradient measurements are necessary. Our gradient descent algorithm
works with the recently introduced zCDP version of differential privacy. It
outperforms prior algorithms for model fitting and is competitive with the
state-of-the-art for $(\epsilon,\delta)$-differential privacy, a strictly
weaker definition than zCDP.
