Evaluating probabilistic classifiers: Reliability diagrams and score
  decompositions revisited
  A probability forecast or probabilistic classifier is reliable or calibrated
if the predicted probabilities are matched by ex post observed frequencies, as
examined visually in reliability diagrams. The classical binning and counting
approach to plotting reliability diagrams has been hampered by a lack of
stability under unavoidable, ad hoc implementation decisions. Here we introduce
the CORP approach, which generates provably statistically Consistent, Optimally
binned, and Reproducible reliability diagrams in an automated way. CORP is
based on non-parametric isotonic regression and implemented via the
Pool-adjacent-violators (PAV) algorithm - essentially, the CORP reliability
diagram shows the graph of the PAV- (re)calibrated forecast probabilities. The
CORP approach allows for uncertainty quantification via either resampling
techniques or asymptotic theory, furnishes a new numerical measure of
miscalibration, and provides a CORP based Brier score decomposition that
generalizes to any proper scoring rule. We anticipate that judicious uses of
the PAV algorithm yield improved tools for diagnostics and inference for a very
wide range of statistical and machine learning methods.
