Efficient and Differentiable Conformal Prediction with General Function
  Classes
  Quantifying the data uncertainty in learning tasks is often done by learning
a prediction interval or prediction set of the label given the input. Two
commonly desired properties for learned prediction sets are \emph{valid
coverage} and \emph{good efficiency} (such as low length or low cardinality).
Conformal prediction is a powerful technique for learning prediction sets with
valid coverage, yet by default its conformalization step only learns a single
parameter, and does not optimize the efficiency over more expressive function
classes.
  In this paper, we propose a generalization of conformal prediction to
multiple learnable parameters, by considering the constrained empirical risk
minimization (ERM) problem of finding the most efficient prediction set subject
to valid empirical coverage. This meta-algorithm generalizes existing conformal
prediction algorithms, and we show that it achieves approximate valid
population coverage and near-optimal efficiency within class, whenever the
function class in the conformalization step is low-capacity in a certain sense.
Next, this ERM problem is challenging to optimize as it involves a
non-differentiable coverage constraint. We develop a gradient-based algorithm
for it by approximating the original constrained ERM using differentiable
surrogate losses and Lagrangians. Experiments show that our algorithm is able
to learn valid prediction sets and improve the efficiency significantly over
existing approaches in several applications such as prediction intervals with
improved length, minimum-volume prediction sets for multi-output regression,
and label prediction sets for image classification.
