Super-convergence and Differential Privacy: Training faster with better
  privacy guarantees
  The combination of deep neural networks and Differential Privacy has been of
increasing interest in recent years, as it offers important data protection
guarantees to the individuals of the training datasets used. However, using
Differential Privacy in the training of neural networks comes with a set of
shortcomings, like a decrease in validation accuracy and a significant increase
in the use of resources and time in training. In this paper, we examine
super-convergence as a way of greatly increasing training speed of
differentially private neural networks, addressing the shortcoming of high
training time and resource use. Super-convergence allows for acceleration in
network training using very high learning rates, and has been shown to achieve
models with high utility in orders of magnitude less training iterations than
conventional ways. Experiments in this paper show that this order-of-magnitude
speedup can also be seen when combining it with Differential Privacy, allowing
for higher validation accuracies in much fewer training iterations compared to
non-private, non-super convergent baseline models. Furthermore,
super-convergence is shown to improve the privacy guarantees of private models.
