On the Calibration of Multiclass Classification with Rejection
  We investigate the problem of multiclass classification with rejection, where
a classifier can choose not to make a prediction to avoid critical
misclassification. First, we consider an approach based on simultaneous
training of a classifier and a rejector, which achieves the state-of-the-art
performance in the binary case. We analyze this approach for the multiclass
case and derive a general condition for calibration to the Bayes-optimal
solution, which suggests that calibration is hard to achieve by general loss
functions unlike the binary case. Next, we consider another traditional
approach based on confidence scores, in which the existing work focuses on a
specific class of losses. We propose rejection criteria for more general losses
for this approach and guarantee calibration to the Bayes-optimal solution.
Finally, we conduct experiments to validate the relevance of our theoretical
findings.
