Revisiting the Evaluation of Uncertainty Estimation and Its Application
  to Explore Model Complexity-Uncertainty Trade-Off
  Accurately estimating uncertainties in neural network predictions is of great
importance in building trusted DNNs-based models, and there is an increasing
interest in providing accurate uncertainty estimation on many tasks, such as
security cameras and autonomous driving vehicles. In this paper, we focus on
the two main use cases of uncertainty estimation, i.e. selective prediction and
confidence calibration. We first reveal potential issues of commonly used
quality metrics for uncertainty estimation in both use cases, and propose our
new metrics to mitigate them. We then apply these new metrics to explore the
trade-off between model complexity and uncertainty estimation quality, a
critically missing work in the literature. Our empirical experiment results
validate the superiority of the proposed metrics, and some interesting trends
about the complexity-uncertainty trade-off are observed.
