Towards Plausible Differentially Private ADMM Based Distributed Machine
  Learning
  The Alternating Direction Method of Multipliers (ADMM) and its distributed
version have been widely used in machine learning. In the iterations of ADMM,
model updates using local private data and model exchanges among agents impose
critical privacy concerns. Despite some pioneering works to relieve such
concerns, differentially private ADMM still confronts many research challenges.
For example, the guarantee of differential privacy (DP) relies on the premise
that the optimality of each local problem can be perfectly attained in each
ADMM iteration, which may never happen in practice. The model trained by DP
ADMM may have low prediction accuracy. In this paper, we address these concerns
by proposing a novel (Improved) Plausible differentially Private ADMM
algorithm, called PP-ADMM and IPP-ADMM. In PP-ADMM, each agent approximately
solves a perturbed optimization problem that is formulated from its local
private data in an iteration, and then perturbs the approximate solution with
Gaussian noise to provide the DP guarantee. To further improve the model
accuracy and convergence, an improved version IPP-ADMM adopts sparse vector
technique (SVT) to determine if an agent should update its neighbors with the
current perturbed solution. The agent calculates the difference of the current
solution from that in the last iteration, and if the difference is larger than
a threshold, it passes the solution to neighbors; or otherwise the solution
will be discarded. Moreover, we propose to track the total privacy loss under
the zero-concentrated DP (zCDP) and provide a generalization performance
analysis. Experiments on real-world datasets demonstrate that under the same
privacy guarantee, the proposed algorithms are superior to the state of the art
in terms of model accuracy and convergence rate.
