Evaluating Scalable Uncertainty Estimation Methods for DNN-Based
  Molecular Property Prediction
  Advances in deep neural network (DNN) based molecular property prediction
have recently led to the development of models of remarkable accuracy and
generalization ability, with graph convolution neural networks (GCNNs)
reporting state-of-the-art performance for this task. However, some challenges
remain and one of the most important that needs to be fully addressed concerns
uncertainty quantification. DNN performance is affected by the volume and the
quality of the training samples. Therefore, establishing when and to what
extent a prediction can be considered reliable is just as important as
outputting accurate predictions, especially when out-of-domain molecules are
targeted. Recently, several methods to account for uncertainty in DNNs have
been proposed, most of which are based on approximate Bayesian inference. Among
these, only a few scale to the large datasets required in applications.
Evaluating and comparing these methods has recently attracted great interest,
but results are generally fragmented and absent for molecular property
prediction. In this paper, we aim to quantitatively compare scalable techniques
for uncertainty estimation in GCNNs. We introduce a set of quantitative
criteria to capture different uncertainty aspects, and then use these criteria
to compare MC-Dropout, deep ensembles, and bootstrapping, both theoretically in
a unified framework that separates aleatoric/epistemic uncertainty and
experimentally on the QM9 dataset. Our experiments quantify the performance of
the different uncertainty estimation methods and their impact on
uncertainty-related error reduction. Our findings indicate that ensembling and
bootstrapping consistently outperform MC-Dropout, with different
context-specific pros and cons. Our analysis also leads to a better
understanding of the role of aleatoric/epistemic uncertainty and highlights the
challenge posed by out-of-domain uncertainty.
