How Do Graph Networks Generalize to Large and Diverse Molecular Systems?
  The predominant method of demonstrating progress of atomic graph neural
networks are benchmarks on small and limited datasets. The implicit hypothesis
behind this approach is that progress on these narrow datasets generalize to
the large diversity of chemistry. This generalizability would be very helpful
for research, but currently remains untested. In this work we test this
assumption by identifying four aspects of complexity in which many datasets are
lacking: 1. Chemical diversity (number of different elements), 2. system size
(number of atoms per sample), 3. dataset size (number of data samples), and 4.
domain shift (similarity of the training and test set). We introduce multiple
subsets of the large Open Catalyst 2020 (OC20) dataset to independently
investigate each of these aspects. We then perform 21 ablation studies and
sensitivity analyses on 9 datasets testing both previously proposed and new
model enhancements. We find that some improvements are consistent between
datasets, but many are not and some even have opposite effects. Based on this
analysis, we identify a smaller dataset that correlates well with the full OC20
dataset, and propose the GemNet-OC model, which outperforms the previous
state-of-the-art on OC20 by 16%, while reducing training time by a factor of
10. Overall, our findings challenge the common belief that graph neural
networks work equally well independent of dataset size and diversity, and
suggest that caution must be exercised when making generalizations based on
narrow datasets.
