Online Multivalid Learning: Means, Moments, and Prediction Intervals
  We present a general, efficient technique for providing contextual
predictions that are "multivalid" in various senses, against an online sequence
of adversarially chosen examples $(x,y)$. This means that the resulting
estimates correctly predict various statistics of the labels $y$ not just
marginally -- as averaged over the sequence of examples -- but also
conditionally on $x \in G$ for any $G$ belonging to an arbitrary intersecting
collection of groups $\mathcal{G}$.
  We provide three instantiations of this framework. The first is mean
prediction, which corresponds to an online algorithm satisfying the notion of
multicalibration from Hebert-Johnson et al. The second is variance and higher
moment prediction, which corresponds to an online algorithm satisfying the
notion of mean-conditioned moment multicalibration from Jung et al. Finally, we
define a new notion of prediction interval multivalidity, and give an algorithm
for finding prediction intervals which satisfy it. Because our algorithms
handle adversarially chosen examples, they can equally well be used to predict
statistics of the residuals of arbitrary point prediction methods, giving rise
to very general techniques for quantifying the uncertainty of predictions of
black box algorithms, even in an online adversarial setting. When instantiated
for prediction intervals, this solves a similar problem as conformal
prediction, but in an adversarial environment and with multivalidity guarantees
stronger than simple marginal coverage guarantees.
