Deep Confidence: A Computationally Efficient Framework for Calculating
  Reliable Errors for Deep Neural Networks
  Deep learning architectures have proved versatile in a number of drug
discovery applications, including the modelling of in vitro compound activity.
While controlling for prediction confidence is essential to increase the trust,
interpretability and usefulness of virtual screening models in drug discovery,
techniques to estimate the reliability of the predictions generated with deep
learning networks remain largely underexplored. Here, we present Deep
Confidence, a framework to compute valid and efficient confidence intervals for
individual predictions using the deep learning technique Snapshot Ensembling
and conformal prediction. Specifically, Deep Confidence generates an ensemble
of deep neural networks by recording the network parameters throughout the
local minima visited during the optimization phase of a single neural network.
This approach serves to derive a set of base learners (i.e., snapshots) with
comparable predictive power on average, that will however generate slightly
different predictions for a given instance. The variability across base
learners and the validation residuals are in turn harnessed to compute
confidence intervals using the conformal prediction framework. Using a set of
24 diverse IC50 data sets from ChEMBL 23, we show that Snapshot Ensembles
perform on par with Random Forest (RF) and ensembles of independently trained
deep neural networks. In addition, we find that the confidence regions
predicted using the Deep Confidence framework span a narrower set of values.
Overall, Deep Confidence represents a highly versatile error prediction
framework that can be applied to any deep learning-based application at no
extra computational cost.
