Private Adaptive Gradient Methods for Convex Optimization
  We study adaptive methods for differentially private convex optimization,
proposing and analyzing differentially private variants of a Stochastic
Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the
AdaGrad algorithm. We provide upper bounds on the regret of both algorithms and
show that the bounds are (worst-case) optimal. As a consequence of our
development, we show that our private versions of AdaGrad outperform adaptive
SGD, which in turn outperforms traditional SGD in scenarios with non-isotropic
gradients where (non-private) Adagrad provably outperforms SGD. The major
challenge is that the isotropic noise typically added for privacy dominates the
signal in gradient geometry for high-dimensional problems; approaches to this
that effectively optimize over lower-dimensional subspaces simply ignore the
actual problems that varying gradient geometries introduce. In contrast, we
study non-isotropic clipping and noise addition, developing a principled
theoretical approach; the consequent procedures also enjoy significantly
stronger empirical performance than prior approaches.
