Federated Model Distillation with Noise-Free Differential Privacy
  Conventional federated learning directly averages model weights, which is
only possible for collaboration between models with homogeneous architectures.
Sharing prediction instead of weight removes this obstacle and eliminates the
risk of white-box inference attacks in conventional federated learning.
However, the predictions from local models are sensitive and would leak
training data privacy to the public. To address this issue, one naive approach
is adding the differentially private random noise to the predictions, which
however brings a substantial trade-off between privacy budget and model
performance. In this paper, we propose a novel framework called FEDMD-NFDP,
which applies a Noise-Free Differential Privacy (NFDP) mechanism into a
federated model distillation framework. Our extensive experimental results on
various datasets validate that FEDMD-NFDP can deliver not only comparable
utility and communication efficiency but also provide a noise-free differential
privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by
considering both IID and non-IID setting, heterogeneous model architectures,
and unlabelled public datasets from a different distribution.
