Efficient Privacy-Preserving Stochastic Nonconvex Optimization
  While many solutions for privacy-preserving convex empirical risk
minimization (ERM) have been developed, privacy-preserving nonconvex ERM
remains a challenge. We study nonconvex ERM, which takes the form of minimizing
a finite-sum of nonconvex loss functions over a training set. We propose a new
differentially private stochastic gradient descent algorithm for nonconvex ERM
that achieves strong privacy guarantees efficiently, and provide a tight
analysis of its privacy and utility guarantees, as well as its gradient
complexity. Our algorithm substantially reduces gradient complexity while
matching the best previous utility guarantee given by Wang et al. (2017). We
extend our algorithm to the distributed setting using secure multi-party
computation, and show it is possible for a distributed algorithm to match the
privacy and utility guarantees of a centralized algorithm in this setting. Our
experiments on benchmark nonconvex ERM problems demonstrate superior
performance in terms of both training cost and utility gains compared with
previous differentially private methods using the same privacy budgets.
