Uncertainty Characteristics Curves: A Systematic Assessment of
  Prediction Intervals
  Accurate quantification of model uncertainty has long been recognized as a
fundamental requirement for trusted AI. In regression tasks, uncertainty is
typically quantified using prediction intervals calibrated to a specific
operating point, making evaluation and comparison across different studies
difficult. Our work leverages: (1) the concept of operating characteristics
curves and (2) the notion of a gain over a simple reference, to derive a novel
operating point agnostic assessment methodology for prediction intervals. The
paper describes the corresponding algorithm, provides a theoretical analysis,
and demonstrates its utility in multiple scenarios. We argue that the proposed
method addresses the current need for comprehensive assessment of prediction
intervals and thus represents a valuable addition to the uncertainty
quantification toolbox.
