Design Space for Graph Neural Networks
  The rapid evolution of Graph Neural Networks (GNNs) has led to a growing
number of new architectures as well as novel applications. However, current
research focuses on proposing and evaluating specific architectural designs of
GNNs, as opposed to studying the more general design space of GNNs that
consists of a Cartesian product of different design dimensions, such as the
number of layers or the type of the aggregation function. Additionally, GNN
designs are often specialized to a single task, yet few efforts have been made
to understand how to quickly find the best GNN design for a novel task or a
novel dataset. Here we define and systematically study the architectural design
space for GNNs which consists of 315,000 different designs over 32 different
predictive tasks. Our approach features three key innovations: (1) A general
GNN design space; (2) a GNN task space with a similarity metric, so that for a
given novel task/dataset, we can quickly identify/transfer the best performing
architecture; (3) an efficient and effective design space evaluation method
which allows insights to be distilled from a huge number of model-task
combinations. Our key results include: (1) A comprehensive set of guidelines
for designing well-performing GNNs; (2) while best GNN designs for different
tasks vary significantly, the GNN task space allows for transferring the best
designs across different tasks; (3) models discovered using our design space
achieve state-of-the-art performance. Overall, our work offers a principled and
scalable approach to transition from studying individual GNN designs for
specific tasks, to systematically studying the GNN design space and the task
space. Finally, we release GraphGym, a powerful platform for exploring
different GNN designs and tasks. GraphGym features modularized GNN
implementation, standardized GNN evaluation, and reproducible and scalable
experiment management.
