Automatic Clipping: Differentially Private Deep Learning Made Easier and
  Stronger
  Per-example gradient clipping is a key algorithmic step that enables
practical differential private (DP) training for deep learning models. The
choice of clipping norm $R$, however, is shown to be vital for achieving high
accuracy under DP. We propose an easy-to-use replacement, called AutoClipping,
that eliminates the need to tune $R$ for any DP optimizers, including DP-SGD,
DP-Adam, DP-LAMB and many others. The automatic variants are as private and
computationally efficient as existing DP optimizers, but require no DP-specific
hyperparameters and thus make DP training as amenable as the standard
non-private training. We give a rigorous convergence analysis of automatic
DP-SGD in the non-convex setting, which shows that it enjoys an asymptotic
convergence rate that matches the standard SGD. We also demonstrate on various
language and vision tasks that automatic clipping outperforms or matches the
state-of-the-art, and can be easily employed with minimal changes to existing
codebases.
