Few-shot Conformal Prediction with Auxiliary Tasks
  We develop a novel approach to conformal prediction when the target task has
limited data available for training. Conformal prediction identifies a small
set of promising output candidates in place of a single prediction, with
guarantees that the set contains the correct answer with high probability. When
training data is limited, however, the predicted set can easily become unusably
large. In this work, we obtain substantially tighter prediction sets while
maintaining desirable marginal guarantees by casting conformal prediction as a
meta-learning paradigm over exchangeable collections of auxiliary tasks. Our
conformalization algorithm is simple, fast, and agnostic to the choice of
underlying model, learning algorithm, or dataset. We demonstrate the
effectiveness of this approach across a number of few-shot classification and
regression tasks in natural language processing, computer vision, and
computational chemistry for drug discovery.
