Data-dependent PAC-Bayes priors via differential privacy
  The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999)
can incorporate knowledge about the learning algorithm and (data) distribution
through the use of distribution-dependent priors, yielding tighter
generalization bounds on data-dependent posteriors. Using this flexibility,
however, is difficult, especially when the data distribution is presumed to be
unknown. We show how an {\epsilon}-differentially private data-dependent prior
yields a valid PAC-Bayes bound, and then show how non-private mechanisms for
choosing priors can also yield generalization bounds. As an application of this
result, we show that a Gaussian prior mean chosen via stochastic gradient
Langevin dynamics (SGLD; Welling and Teh, 2011) leads to a valid PAC-Bayes
bound given control of the 2-Wasserstein distance to an
{\epsilon}-differentially private stationary distribution. We study our
data-dependent bounds empirically, and show that they can be nonvacuous even
when other distribution-dependent bounds are vacuous.
