Decentralized Stochastic Optimization with Inherent Privacy Protection
  Decentralized stochastic optimization is the basic building block of modern
collaborative machine learning, distributed estimation and control, and
large-scale sensing. Since involved data usually contain sensitive information
like user locations, healthcare records and financial transactions, privacy
protection has become an increasingly pressing need in the implementation of
decentralized stochastic optimization algorithms. In this paper, we propose a
decentralized stochastic gradient descent algorithm which is embedded with
inherent privacy protection for every participating agent against other
participating agents and external eavesdroppers. This proposed algorithm builds
in a dynamics based gradient-obfuscation mechanism to enable privacy protection
without compromising optimization accuracy, which is in significant difference
from differential-privacy based privacy solutions for decentralized
optimization that have to trade optimization accuracy for privacy. The dynamics
based privacy approach is encryption-free, and hence avoids incurring heavy
communication or computation overhead, which is a common problem with
encryption based privacy solutions for decentralized stochastic optimization.
Besides rigorously characterizing the convergence performance of the proposed
decentralized stochastic gradient descent algorithm under both convex objective
functions and non-convex objective functions, we also provide rigorous
information-theoretic analysis of its strength of privacy protection.
Simulation results for a distributed estimation problem as well as numerical
experiments for decentralized learning on a benchmark machine learning dataset
confirm the effectiveness of the proposed approach.
