BERT Learns (and Teaches) Chemistry
  Modern computational organic chemistry is becoming increasingly data-driven.
There remain a large number of important unsolved problems in this area such as
product prediction given reactants, drug discovery, and metric-optimized
molecule synthesis, but efforts to solve these problems using machine learning
have also increased in recent years. In this work, we propose the use of
attention to study functional groups and other property-impacting molecular
substructures from a data-driven perspective, using a transformer-based model
(BERT) on datasets of string representations of molecules and analyzing the
behavior of its attention heads. We then apply the representations of
functional groups and atoms learned by the model to tackle problems of
toxicity, solubility, drug-likeness, and synthesis accessibility on smaller
datasets using the learned representations as features for graph convolution
and attention models on the graph structure of molecules, as well as
fine-tuning of BERT. Finally, we propose the use of attention visualization as
a helpful tool for chemistry practitioners and students to quickly identify
important substructures in various chemical properties.
