Communication Efficient Federated Learning via Ordered ADMM in a Fully
  Decentralized Setting
  The challenge of communication-efficient distributed optimization has
attracted attention in recent years. In this paper, a communication efficient
algorithm, called ordering-based alternating direction method of multipliers
(OADMM) is devised in a general fully decentralized network setting where a
worker can only exchange messages with neighbors. Compared to the classical
ADMM, a key feature of OADMM is that transmissions are ordered among workers at
each iteration such that a worker with the most informative data broadcasts its
local variable to neighbors first, and neighbors who have not transmitted yet
can update their local variables based on that received transmission. In OADMM,
we prohibit workers from transmitting if their current local variables are not
sufficiently different from their previously transmitted value. A variant of
OADMM, called SOADMM, is proposed where transmissions are ordered but
transmissions are never stopped for each node at each iteration. Numerical
results demonstrate that given a targeted accuracy, OADMM can significantly
reduce the number of communications compared to existing algorithms including
ADMM. We also show numerically that SOADMM can accelerate convergence,
resulting in communication savings compared to the classical ADMM.
