Benchmarking Graph Neural Networks
  In the last few years, graph neural networks (GNNs) have become the standard
toolkit for analyzing and learning from data on graphs. This emerging field has
witnessed an extensive growth of promising techniques that have been applied
with success to computer science, mathematics, biology, physics and chemistry.
But for any successful field to become mainstream and reliable, benchmarks must
be developed to quantify progress. This led us in March 2020 to release a
benchmark framework that i) comprises of a diverse collection of mathematical
and real-world graphs, ii) enables fair model comparison with the same
parameter budget to identify key architectures, iii) has an open-source,
easy-to-use and reproducible code infrastructure, and iv) is flexible for
researchers to experiment with new theoretical ideas. As of May 2022, the
GitHub repository has reached 1,800 stars and 339 forks, which demonstrates the
utility of the proposed open-source framework through the wide usage by the GNN
community. In this paper, we present an updated version of our benchmark with a
concise presentation of the aforementioned framework characteristics, an
additional medium-sized molecular dataset AQSOL, similar to the popular ZINC,
but with a real-world measured chemical target, and discuss how this framework
can be leveraged to explore new GNN designs and insights. As a proof of value
of our benchmark, we study the case of graph positional encoding (PE) in GNNs,
which was introduced with this benchmark and has since spurred interest of
exploring more powerful PE for Transformers and GNNs in a robust experimental
setting.
