Towards Efficient and Stable K-Asynchronous Federated Learning with
  Unbounded Stale Gradients on Non-IID Data
  Federated learning (FL) is an emerging privacy-preserving paradigm that
enables multiple participants collaboratively to train a global model without
uploading raw data. Considering heterogeneous computing and communication
capabilities of different participants, asynchronous FL can avoid the
stragglers effect in synchronous FL and adapts to scenarios with vast
participants. Both staleness and non-IID data in asynchronous FL would reduce
the model utility. However, there exists an inherent contradiction between the
solutions to the two problems. That is, mitigating the staleness requires to
select less but consistent gradients while coping with non-IID data demands
more comprehensive gradients. To address the dilemma, this paper proposes a
two-stage weighted $K$ asynchronous FL with adaptive learning rate (WKAFL). By
selecting consistent gradients and adjusting learning rate adaptively, WKAFL
utilizes stale gradients and mitigates the impact of non-IID data, which can
achieve multifaceted enhancement in training speed, prediction accuracy and
training stability. We also present the convergence analysis for WKAFL under
the assumption of unbounded staleness to understand the impact of staleness and
non-IID data. Experiments implemented on both benchmark and synthetic FL
datasets show that WKAFL has better overall performance compared to existing
algorithms.
