Layer-Dependent Importance Sampling for Training Deep and Large Graph
  Convolutional Networks
  Graph convolutional networks (GCNs) have recently received wide attentions,
due to their successful applications in different graph tasks and different
domains. Training GCNs for a large graph, however, is still a challenge.
Original full-batch GCN training requires calculating the representation of all
the nodes in the graph per GCN layer, which brings in high computation and
memory costs. To alleviate this issue, several sampling-based methods have been
proposed to train GCNs on a subset of nodes. Among them, the node-wise
neighbor-sampling method recursively samples a fixed number of neighbor nodes,
and thus its computation cost suffers from exponential growing neighbor size;
while the layer-wise importance-sampling method discards the neighbor-dependent
constraints, and thus the nodes sampled across layer suffer from sparse
connection problem. To deal with the above two problems, we propose a new
effective sampling algorithm called LAyer-Dependent ImportancE Sampling
(LADIES). Based on the sampled nodes in the upper layer, LADIES selects their
neighborhood nodes, constructs a bipartite subgraph and computes the importance
probability accordingly. Then, it samples a fixed number of nodes by the
calculated probability, and recursively conducts such procedure per layer to
construct the whole computation graph. We prove theoretically and
experimentally, that our proposed sampling algorithm outperforms the previous
sampling methods in terms of both time and memory costs. Furthermore, LADIES is
shown to have better generalization accuracy than original full-batch GCN, due
to its stochastic nature.
