Training Uncertainty-Aware Classifiers with Conformalized Deep Learning
Deep neural networks are powerful tools to detect hidden patterns in data and
leverage them to make predictions, but they are not designed to understand
uncertainty and estimate reliable probabilities. In particular, they tend to be
overconfident. We address this problem by developing a novel training algorithm
that can lead to more dependable uncertainty estimates, without sacrificing
predictive power. The idea is to mitigate overconfidence by minimizing a loss
function, inspired by advances in conformal inference, that quantifies model
uncertainty by carefully leveraging hold-out data. Experiments with synthetic
and real data demonstrate this method leads to smaller conformal prediction
sets with higher conditional coverage, after exact calibration with hold-out
data, compared to state-of-the-art alternatives.