Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for
  Private Learning
  The privacy leakage of the model about the training data can be bounded in
the differential privacy mechanism. However, for meaningful privacy parameters,
a differentially private model degrades the utility drastically when the model
comprises a large number of trainable parameters. In this paper, we propose an
algorithm \emph{Gradient Embedding Perturbation (GEP)} towards training
differentially private deep models with decent accuracy. Specifically, in each
gradient descent step, GEP first projects individual private gradient into a
non-sensitive anchor subspace, producing a low-dimensional gradient embedding
and a small-norm residual gradient. Then, GEP perturbs the low-dimensional
embedding and the residual gradient separately according to the privacy budget.
Such a decomposition permits a small perturbation variance, which greatly helps
to break the dimensional barrier of private learning. With GEP, we achieve
decent accuracy with reasonable computational cost and modest privacy guarantee
for deep models. Especially, with privacy bound $\epsilon=8$, we achieve
$74.9\%$ test accuracy on CIFAR10 and $95.1\%$ test accuracy on SVHN,
significantly improving over existing results.
