Federated Deep Learning with Bayesian Privacy
  Federated learning (FL) aims to protect data privacy by cooperatively
learning a model without sharing private data among users. For Federated
Learning of Deep Neural Network with billions of model parameters, existing
privacy-preserving solutions are unsatisfactory. Homomorphic encryption (HE)
based methods provide secure privacy protections but suffer from extremely high
computational and communication overheads rendering it almost useless in
practice . Deep learning with Differential Privacy (DP) was implemented as a
practical learning algorithm at a manageable cost in complexity. However, DP is
vulnerable to aggressive Bayesian restoration attacks as disclosed in the
literature and demonstrated in experimental results of this work. To address
the aforementioned perplexity, we propose a novel Bayesian Privacy (BP)
framework which enables Bayesian restoration attacks to be formulated as the
probability of reconstructing private data from observed public information.
Specifically, the proposed BP framework accurately quantifies privacy loss by
Kullback-Leibler (KL) Divergence between the prior distribution about the
privacy data and the posterior distribution of restoration private data
conditioning on exposed information}. To our best knowledge, this Bayesian
Privacy analysis is the first to provides theoretical justification of secure
privacy-preserving capabilities against Bayesian restoration attacks. As a
concrete use case, we demonstrate that a novel federated deep learning method
using private passport layers is able to simultaneously achieve high model
performance, privacy-preserving capability and low computational complexity.
Theoretical analysis is in accordance with empirical measurements of
information leakage extensively experimented with a variety of DNN networks on
image classification MNIST, CIFAR10, and CIFAR100 datasets.
