Learn Locally, Correct Globally: A Distributed Algorithm for Training
  Graph Neural Networks
  Despite the recent success of Graph Neural Networks (GNNs), training GNNs on
large graphs remains challenging. The limited resource capacities of the
existing servers, the dependency between nodes in a graph, and the privacy
concern due to the centralized storage and model learning have spurred the need
to design an effective distributed algorithm for GNN training. However,
existing distributed GNN training methods impose either excessive communication
costs or large memory overheads that hinders their scalability. To overcome
these issues, we propose a communication-efficient distributed GNN training
technique named $\text{{Learn Locally, Correct Globally}}$ (LLCG). To reduce
the communication and memory overhead, each local machine in LLCG first trains
a GNN on its local data by ignoring the dependency between nodes among
different machines, then sends the locally trained model to the server for
periodic model averaging. However, ignoring node dependency could result in
significant performance degradation. To solve the performance degradation, we
propose to apply $\text{{Global Server Corrections}}$ on the server to refine
the locally learned models. We rigorously analyze the convergence of
distributed methods with periodic model averaging for training GNNs and show
that naively applying periodic model averaging but ignoring the dependency
between nodes will suffer from an irreducible residual error. However, this
residual error can be eliminated by utilizing the proposed global corrections
to entail fast convergence rate. Extensive experiments on real-world datasets
show that LLCG can significantly improve the efficiency without hurting the
performance.
