Learning Optimal Conformal Classifiers
  Modern deep learning based classifiers show very high accuracy on test data
but this does not provide sufficient guarantees for safe deployment, especially
in high-stake AI applications such as medical diagnosis. Usually, predictions
are obtained without a reliable uncertainty estimate or a formal guarantee.
Conformal prediction (CP) addresses these issues by using the classifier's
predictions, e.g., its probability estimates, to predict confidence sets
containing the true class with a user-specified probability. However, using CP
as a separate processing step after training prevents the underlying model from
adapting to the prediction of confidence sets. Thus, this paper explores
strategies to differentiate through CP during training with the goal of
training model with the conformal wrapper end-to-end. In our approach,
conformal training (ConfTr), we specifically "simulate" conformalization on
mini-batches during training. Compared to standard training, ConfTr reduces the
average confidence set size (inefficiency) of state-of-the-art CP methods
applied after training. Moreover, it allows to "shape" the confidence sets
predicted at test time, which is difficult for standard CP. On experiments with
several datasets, we show ConfTr can influence how inefficiency is distributed
across classes, or guide the composition of confidence sets in terms of the
included classes, while retaining the guarantees offered by CP.
