Communication-Efficient Sampling for Distributed Training of Graph
  Convolutional Networks
  Training Graph Convolutional Networks (GCNs) is expensive as it needs to
aggregate data recursively from neighboring nodes. To reduce the computation
overhead, previous works have proposed various neighbor sampling methods that
estimate the aggregation result based on a small number of sampled neighbors.
Although these methods have successfully accelerated the training, they mainly
focus on the single-machine setting. As real-world graphs are large, training
GCNs in distributed systems is desirable. However, we found that the existing
neighbor sampling methods do not work well in a distributed setting.
Specifically, a naive implementation may incur a huge amount of communication
of feature vectors among different machines. To address this problem, we
propose a communication-efficient neighbor sampling method in this work. Our
main idea is to assign higher sampling probabilities to the local nodes so that
remote nodes are accessed less frequently. We present an algorithm that
determines the local sampling probabilities and makes sure our skewed neighbor
sampling does not affect much the convergence of the training. Our experiments
with node classification benchmarks show that our method significantly reduces
the communication overhead for distributed GCN training with little accuracy
loss.
