Weighted Distributed Differential Privacy ERM: Convex and Non-convex
  Distributed machine learning is an approach allowing different parties to
learn a model over all data sets without disclosing their own data. In this
paper, we propose a weighted distributed differential privacy (WD-DP) empirical
risk minimization (ERM) method to train a model in distributed setting,
considering different weights of different clients. We guarantee differential
privacy by gradient perturbation, adding Gaussian noise, and advance the
state-of-the-art on gradient perturbation method in distributed setting. By
detailed theoretical analysis, we show that in distributed setting, the noise
bound and the excess empirical risk bound can be improved by considering
different weights held by multiple parties. Moreover, considering that the
constraint of convex loss function in ERM is not easy to achieve in some
situations, we generalize our method to non-convex loss functions which satisfy
Polyak-Lojasiewicz condition. Experiments on real data sets show that our
method is more reliable and we improve the performance of distributed
differential privacy ERM, especially in the case that data scale on different
clients is uneven.
