Gradient Perturbation is Underrated for Differentially Private Convex
  Optimization
  Gradient perturbation, widely used for differentially private optimization,
injects noise at every iterative update to guarantee differential privacy.
Previous work first determines the noise level that can satisfy the privacy
requirement and then analyzes the utility of noisy gradient updates as in the
non-private case. In contrast, we explore how privacy noise affects
optimization property. We show that for differentially private convex
optimization, the utility guarantee of differentially private (stochastic)
gradient descent is determined by an \emph{expected curvature} rather than the
minimum curvature. The \emph{expected curvature}, which represents the average
curvature over the optimization path, is usually much larger than the minimum
curvature. By using the \emph{expected curvature}, we show that gradient
perturbation can achieve a significantly improved utility guarantee that can
theoretically justify the advantage of gradient perturbation over other
perturbation methods. Finally, our extensive experiments suggest that gradient
perturbation with the advanced composition method indeed outperforms other
perturbation approaches by a large margin, matching our theoretical findings.
