Towards Efficient Large-Scale Graph Neural Network Computing
  Recent deep learning models have moved beyond low-dimensional regular grids
such as image, video, and speech, to high-dimensional graph-structured data,
such as social networks, brain connections, and knowledge graphs. This
evolution has led to large graph-based irregular and sparse models that go
beyond what existing deep learning frameworks are designed for. Further, these
models are not easily amenable to efficient, at scale, acceleration on parallel
hardwares (e.g. GPUs). We introduce NGra, the first parallel processing
framework for graph-based deep neural networks (GNNs). NGra presents a new
SAGA-NN model for expressing deep neural networks as vertex programs with each
layer in well-defined (Scatter, ApplyEdge, Gather, ApplyVertex) graph operation
stages. This model not only allows GNNs to be expressed intuitively, but also
facilitates the mapping to an efficient dataflow representation. NGra addresses
the scalability challenge transparently through automatic graph partitioning
and chunk-based stream processing out of GPU core or over multiple GPUs, which
carefully considers data locality, data movement, and overlapping of parallel
processing and data movement. NGra further achieves efficiency through highly
optimized Scatter/Gather operators on GPUs despite its sparsity. Our evaluation
shows that NGra scales to large real graphs that none of the existing
frameworks can handle directly, while achieving up to about 4 times speedup
even at small scales over the multiple-baseline design on TensorFlow.
