Understanding GNN Computational Graph: A Coordinated Computation, IO,
  and Memory Perspective
  Graph Neural Networks (GNNs) have been widely used in various domains, and
GNNs with sophisticated computational graph lead to higher latency and larger
memory consumption. Optimizing the GNN computational graph suffers from: (1)
Redundant neural operator computation. The same data are propagated through the
graph structure to perform the same neural operation multiple times in GNNs,
leading to redundant computation which accounts for 92.4% of total operators.
(2) Inconsistent thread mapping. Efficient thread mapping schemes for
vertex-centric and edge-centric operators are different. This inconsistency
prohibits operator fusion to reduce memory IO. (3) Excessive intermediate data.
For GNN training which is usually performed concurrently with inference,
intermediate data must be stored for the backward pass, consuming 91.9% of the
total memory requirement. To tackle these challenges, we propose following
designs to optimize the GNN computational graph from a novel coordinated
computation, IO, and memory perspective: (1) Propagation-postponed operator
reorganization. We reorganize operators to perform neural operations before the
propagation, thus the redundant computation is eliminated. (2) Unified thread
mapping for fusion. We propose a unified thread mapping scheme for both vertex-
and edge-centric operators to enable fusion and reduce IO. (3) Intermediate
data recomputation. Intermediate data are recomputed during the backward pass
to reduce the total memory consumption. Extensive experimental results on three
typical GNN models show that, we achieve up to 2.75x end-to-end speedup, 6.89x
less memory IO, and 7.73x less memory consumption over state-of-the-art
frameworks.
