Toward Training at ImageNet Scale with Differential Privacy
  Differential privacy (DP) is the de facto standard for training machine
learning (ML) models, including neural networks, while ensuring the privacy of
individual examples in the training set. Despite a rich literature on how to
train ML models with differential privacy, it remains extremely challenging to
train real-life, large neural networks with both reasonable accuracy and
privacy.
  We set out to investigate how to do this, using ImageNet image classification
as a poster example of an ML task that is very challenging to resolve
accurately with DP right now. This paper shares initial lessons from our
effort, in the hope that it will inspire and inform other researchers to
explore DP training at scale. We show approaches that help make DP training
faster, as well as model types and settings of the training process that tend
to work better in the DP setting. Combined, the methods we discuss let us train
a Resnet-18 with DP to $47.9\%$ accuracy and privacy parameters $\epsilon = 10,
\delta = 10^{-6}$. This is a significant improvement over "naive" DP training
of ImageNet models, but a far cry from the $75\%$ accuracy that can be obtained
by the same network without privacy. The model we use was pretrained on the
Places365 data set as a starting point. We share our code at
https://github.com/google-research/dp-imagenet, calling for others to build
upon this new baseline to further improve DP at scale.
