An Efficient DP-SGD Mechanism for Large Scale NLP Models
  Recent advances in deep learning have drastically improved performance on
many Natural Language Understanding (NLU) tasks. However, the data used to
train NLU models may contain private information such as addresses or phone
numbers, particularly when drawn from human subjects. It is desirable that
underlying models do not expose private information contained in the training
data. Differentially Private Stochastic Gradient Descent (DP-SGD) has been
proposed as a mechanism to build privacy-preserving models. However, DP-SGD can
be prohibitively slow to train. In this work, we propose a more efficient
DP-SGD for training using a GPU infrastructure and apply it to fine-tuning
models based on LSTM and transformer architectures. We report faster training
times, alongside accuracy, theoretical privacy guarantees and success of
Membership inference attacks for our models and observe that fine-tuning with
proposed variant of DP-SGD can yield competitive models without significant
degradation in training time and improvement in privacy protection. We also
make observations such as looser theoretical $\epsilon, \delta$ can translate
into significant practical privacy gains.
