Differentially Private Data Generation Needs Better Features
  Training even moderately-sized generative models with differentially-private
stochastic gradient descent (DP-SGD) is difficult: the required level of noise
for reasonable levels of privacy is simply too large. We advocate instead
building off a good, relevant representation on public data, then using private
data only for "transfer learning." In particular, we minimize the maximum mean
discrepancy (MMD) between private target data and the generated distribution,
using a kernel based on perceptual features from a public dataset. With the
MMD, we can simply privatize the data-dependent term once and for all, rather
than introducing noise at each step of optimization as in DP-SGD. Our algorithm
allows us to generate CIFAR10-level images faithfully with $\varepsilon \approx
2$, far surpassing the current state of the art, which only models MNIST and
FashionMNIST at $\varepsilon \approx 10$. Our work introduces simple yet
powerful foundations for reducing the gap between private and non-private deep
generative models.
