Locally Differentially Private Distributed Deep Learning via Knowledge
  Distillation
  Deep learning often requires a large amount of data. In real-world
applications, e.g., healthcare applications, the data collected by a single
organization (e.g., hospital) is often limited, and the majority of massive and
diverse data is often segregated across multiple organizations. As such, it
motivates the researchers to conduct distributed deep learning, where the data
user would like to build DL models using the data segregated across multiple
different data owners. However, this could lead to severe privacy concerns due
to the sensitive nature of the data, thus the data owners would be hesitant and
reluctant to participate. We propose LDP-DL, a privacy-preserving distributed
deep learning framework via local differential privacy and knowledge
distillation, where each data owner learns a teacher model using its own
(local) private dataset, and the data user learns a student model to mimic the
output of the ensemble of the teacher models. In the experimental evaluation, a
comprehensive comparison has been made among our proposed approach (i.e.,
LDP-DL), DP-SGD, PATE and DP-FL, using three popular deep learning benchmark
datasets (i.e., CIFAR10, MNIST and FashionMNIST). The experimental results show
that LDP-DL consistently outperforms the other competitors in terms of privacy
budget and model accuracy.
