On the Interpretability of Conditional Probability Estimates in the
  Agnostic Setting
  We study the interpretability of conditional probability estimates for binary
classification under the agnostic setting or scenario. Under the agnostic
setting, conditional probability estimates do not necessarily reflect the true
conditional probabilities. Instead, they have a certain calibration property:
among all data points that the classifier has predicted P(Y = 1|X) = p, p
portion of them actually have label Y = 1. For cost-sensitive decision
problems, this calibration property provides adequate support for us to use
Bayes Decision Theory. In this paper, we define a novel measure for the
calibration property together with its empirical counterpart, and prove an
uniform convergence result between them. This new measure enables us to
formally justify the calibration property of conditional probability
estimations, and provides new insights on the problem of estimating and
calibrating conditional probabilities.
