Validating the Validation: Reanalyzing a large-scale comparison of Deep
  Learning and Machine Learning models for bioactivity prediction
  Machine learning methods may have the potential to significantly accelerate
drug discovery. However, the increasing rate of new methodological approaches
being published in the literature raises the fundamental question of how models
should be benchmarked and validated. We reanalyze the data generated by a
recently published large-scale comparison of machine learning models for
bioactivity prediction and arrive at a somewhat different conclusion. We show
that the performance of support vector machines is competitive with that of
deep learning methods. Additionally, using a series of numerical experiments,
we question the relevance of area under the receiver operating characteristic
curve as a metric in virtual screening, and instead suggest that area under the
precision-recall curve should be used in conjunction with the receiver
operating characteristic. Our numerical experiments also highlight challenges
in estimating the uncertainty in model performance via scaffold-split nested
cross validation.
