Finite-Time Consensus Learning for Decentralized Optimization with
  Nonlinear Gossiping
  Distributed learning has become an integral tool for scaling up machine
learning and addressing the growing need for data privacy. Although more robust
to the network topology, decentralized learning schemes have not gained the
same level of popularity as their centralized counterparts for being less
competitive performance-wise. In this work, we attribute this issue to the lack
of synchronization among decentralized learning workers, showing both
empirically and theoretically that the convergence rate is tied to the
synchronization level among the workers. Such motivated, we present a novel
decentralized learning framework based on nonlinear gossiping (NGO), that
enjoys an appealing finite-time consensus property to achieve better
synchronization. We provide a careful analysis of its convergence and discuss
its merits for modern distributed optimization applications, such as deep
neural networks. Our analysis on how communication delay and randomized chats
affect learning further enables the derivation of practical variants that
accommodate asynchronous and randomized communications. To validate the
effectiveness of our proposal, we benchmark NGO against competing solutions
through an extensive set of tests, with encouraging results reported.
