Differentially Private Dropout
  Large data collections required for the training of neural networks often
contain sensitive information such as the medical histories of patients, and
the privacy of the training data must be preserved. In this paper, we introduce
a dropout technique that provides an elegant Bayesian interpretation to
dropout, and show that the intrinsic noise added, with the primary goal of
regularization, can be exploited to obtain a degree of differential privacy.
The iterative nature of training neural networks presents a challenge for
privacy-preserving estimation since multiple iterations increase the amount of
noise added. We overcome this by using a relaxed notion of differential
privacy, called concentrated differential privacy, which provides tighter
estimates on the overall privacy loss. We demonstrate the accuracy of our
privacy-preserving dropout algorithm on benchmark datasets.
