BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks
  with Partition-Parallelism and Random Boundary Node Sampling
  Graph Convolutional Networks (GCNs) have emerged as the state-of-the-art
method for graph-based learning tasks. However, training GCNs at scale is still
challenging, hindering both the exploration of more sophisticated GCN
architectures and their applications to real-world large graphs. While it might
be natural to consider graph partition and distributed training for tackling
this challenge, this direction has only been slightly scratched the surface in
the previous works due to the limitations of existing designs. In this work, we
first analyze why distributed GCN training is ineffective and identify the
underlying cause to be the excessive number of boundary nodes of each
partitioned subgraph, which easily explodes the memory and communication costs
for GCN training. Furthermore, we propose a simple yet effective method dubbed
BNS-GCN that adopts random Boundary-Node-Sampling to enable efficient and
scalable distributed GCN training. Experiments and ablation studies
consistently validate the effectiveness of BNS-GCN, e.g., boosting the
throughput by up to 16.2x and reducing the memory usage by up to 58%, while
maintaining a full-graph accuracy. Furthermore, both theoretical and empirical
analysis show that BNS-GCN enjoys a better convergence than existing
sampling-based methods. We believe that our BNS-GCN has opened up a new
paradigm for enabling GCN training at scale. The code is available at
https://github.com/RICE-EIC/BNS-GCN.
