Receiver operating characteristic (ROC) movies, universal ROC (UROC)
  curves, and coefficient of predictive ability (CPA)
  Throughout science and technology, receiver operating characteristic (ROC)
curves and associated area under the curve (AUC) measures constitute powerful
tools for assessing the predictive abilities of features, markers and tests in
binary classification problems. Despite its immense popularity, ROC analysis
has been subject to a fundamental restriction, in that it applies to
dichotomous (yes or no) outcomes only. Here we introduce ROC movies and
universal ROC (UROC) curves that apply to just any linearly ordered outcome,
along with an associated coefficient of predictive ability (CPA) measure. CPA
equals the area under the UROC curve, and admits appealing interpretations in
terms of probabilities and rank based covariances. For binary outcomes CPA
equals AUC, and for pairwise distinct outcomes CPA relates linearly to
Spearman's coefficient, in the same way that the C index relates linearly to
Kendall's coefficient. ROC movies, UROC curves, and CPA nest and generalize the
tools of classical ROC analysis, and are bound to supersede them in a wealth of
applications. Their usage is illustrated in data examples from biomedicine and
meteorology, where rank based measures yield new insights in the WeatherBench
comparison of the predictive performance of convolutional neural networks and
physical-numerical models for weather prediction.
