Calibrated and Sharp Uncertainties in Deep Learning via Simple Density
  Estimation
  Predictive uncertainties can be characterized by two properties--calibration
and sharpness. This paper argues for reasoning about uncertainty in terms these
properties and proposes simple algorithms for enforcing them in deep learning.
Our methods focus on the strongest notion of calibration--distribution
calibration--and enforce it by fitting a low-dimensional density or quantile
function with a neural estimator. The resulting approach is much simpler and
more broadly applicable than previous methods across both classification and
regression. Empirically, we find that our methods improve predictive
uncertainties on several tasks with minimal computational and implementation
overhead. Our insights suggest simple and improved ways of training deep
learning models that lead to accurate uncertainties that should be leveraged to
improve performance across downstream applications.
