Self-Supervised Graph Transformer on Large-Scale Molecular Data
  How to obtain informative representations of molecules is a crucial
prerequisite in AI-driven drug design and discovery. Recent researches abstract
molecules as graphs and employ Graph Neural Networks (GNNs) for molecular
representation learning. Nevertheless, two issues impede the usage of GNNs in
real scenarios: (1) insufficient labeled molecules for supervised training; (2)
poor generalization capability to new-synthesized molecules. To address them
both, we propose a novel framework, GROVER, which stands for Graph
Representation frOm self-superVised mEssage passing tRansformer. With carefully
designed self-supervised tasks in node-, edge- and graph-level, GROVER can
learn rich structural and semantic information of molecules from enormous
unlabelled molecular data. Rather, to encode such complex information, GROVER
integrates Message Passing Networks into the Transformer-style architecture to
deliver a class of more expressive encoders of molecules. The flexibility of
GROVER allows it to be trained efficiently on large-scale molecular dataset
without requiring any supervision, thus being immunized to the two issues
mentioned above. We pre-train GROVER with 100 million parameters on 10 million
unlabelled molecules -- the biggest GNN and the largest training dataset in
molecular representation learning. We then leverage the pre-trained GROVER for
molecular property prediction followed by task-specific fine-tuning, where we
observe a huge improvement (more than 6% on average) from current
state-of-the-art methods on 11 challenging benchmarks. The insights we gained
are that well-designed self-supervision losses and largely-expressive
pre-trained models enjoy the significant potential on performance boosting.
