CRUDE: Calibrating Regression Uncertainty Distributions Empirically
  Calibrated uncertainty estimates in machine learning are crucial to many
fields such as autonomous vehicles, medicine, and weather and climate
forecasting. While there is extensive literature on uncertainty calibration for
classification, the classification findings do not always translate to
regression. As a result, modern models for predicting uncertainty in regression
settings typically produce uncalibrated and overconfident estimates. To address
these gaps, we present a calibration method for regression settings that does
not assume a particular uncertainty distribution over the error: Calibrating
Regression Uncertainty Distributions Empirically (CRUDE). CRUDE makes the
weaker assumption that error distributions have a constant arbitrary shape
across the output space, shifted by predicted mean and scaled by predicted
standard deviation. We detail a theoretical connection between CRUDE and
conformal inference. Across an extensive set of regression tasks, CRUDE
demonstrates consistently sharper, better calibrated, and more accurate
uncertainty estimates than state-of-the-art techniques.
