Asynchronous Parallel Incremental Block-Coordinate Descent for
  Decentralized Machine Learning
  Machine learning (ML) is a key technique for big-data-driven modelling and
analysis of massive Internet of Things (IoT) based intelligent and ubiquitous
computing. For fast-increasing applications and data amounts, distributed
learning is a promising emerging paradigm since it is often impractical or
inefficient to share/aggregate data to a centralized location from distinct
ones. This paper studies the problem of training an ML model over decentralized
systems, where data are distributed over many user devices and the learning
algorithm run on-device, with the aim of relaxing the burden at a central
entity/server. Although gossip-based approaches have been used for this purpose
in different use cases, they suffer from high communication costs, especially
when the number of devices is large. To mitigate this, incremental-based
methods are proposed. We first introduce incremental block-coordinate descent
(I-BCD) for the decentralized ML, which can reduce communication costs at the
expense of running time. To accelerate the convergence speed, an asynchronous
parallel incremental BCD (API-BCD) method is proposed, where multiple
devices/agents are active in an asynchronous fashion. We derive convergence
properties for the proposed methods. Simulation results also show that our
API-BCD method outperforms state of the art in terms of running time and
communication costs.
