Private Deep Learning with Teacher Ensembles
  Privacy-preserving deep learning is crucial for deploying deep neural network
based solutions, especially when the model works on data that contains
sensitive information. Most privacy-preserving methods lead to undesirable
performance degradation. Ensemble learning is an effective way to improve model
performance. In this work, we propose a new method for teacher ensembles that
uses more informative network outputs under differential private stochastic
gradient descent and provide provable privacy guarantees. Out method employs
knowledge distillation and hint learning on intermediate representations to
facilitate the training of student model. Additionally, we propose a simple
weighted ensemble scheme that works more robustly across different teaching
settings. Experimental results on three common image datasets benchmark (i.e.,
CIFAR10, MINST, and SVHN) demonstrate that our approach outperforms previous
state-of-the-art methods on both performance and privacy-budget.
