Calibration tests in multi-class classification: A unifying framework
  In safety-critical applications a probabilistic model is usually required to
be calibrated, i.e., to capture the uncertainty of its predictions accurately.
In multi-class classification, calibration of the most confident predictions
only is often not sufficient. We propose and study calibration measures for
multi-class classification that generalize existing measures such as the
expected calibration error, the maximum calibration error, and the maximum mean
calibration error. We propose and evaluate empirically different consistent and
unbiased estimators for a specific class of measures based on matrix-valued
kernels. Importantly, these estimators can be interpreted as test statistics
associated with well-defined bounds and approximations of the p-value under the
null hypothesis that the model is calibrated, significantly improving the
interpretability of calibration measures, which otherwise lack any meaningful
unit or scale.
