Calibrating sufficiently
  When probabilistic classifiers are trained and calibrated, the so-called
grouping loss component of the calibration loss can easily be overlooked.
Grouping loss refers to the gap between observable information and information
actually exploited in the calibration exercise. We investigate the relation
between grouping loss and the concept of sufficiency, identifying
comonotonicity as a useful criterion for sufficiency. We revisit the probing
reduction approach of Langford & Zadrozny (2005) and find that it produces an
estimator of probabilistic classifiers that reduces grouping loss. Finally, we
discuss Brier curves as tools to support training and 'sufficient' calibration
of probabilistic classifiers.
