Moment Multicalibration for Uncertainty Estimation
  We show how to achieve the notion of "multicalibration" from H\'ebert-Johnson
et al. [2018] not just for means, but also for variances and other higher
moments. Informally, it means that we can find regression functions which,
given a data point, can make point predictions not just for the expectation of
its label, but for higher moments of its label distribution as well-and those
predictions match the true distribution quantities when averaged not just over
the population as a whole, but also when averaged over an enormous number of
finely defined subgroups. It yields a principled way to estimate the
uncertainty of predictions on many different subgroups-and to diagnose
potential sources of unfairness in the predictive power of features across
subgroups. As an application, we show that our moment estimates can be used to
derive marginal prediction intervals that are simultaneously valid as averaged
over all of the (sufficiently large) subgroups for which moment
multicalibration has been obtained.
