Estimation of Accurate and Calibrated Uncertainties in Deterministic
  models
  In this paper we focus on the problem of assigning uncertainties to
single-point predictions generated by a deterministic model that outputs a
continuous variable. This problem applies to any state-of-the-art physics or
engineering models that have a computational cost that does not readily allow
to run ensembles and to estimate the uncertainty associated to single-point
predictions. Essentially, we devise a method to easily transform a
deterministic prediction into a probabilistic one. We show that for doing so,
one has to compromise between the accuracy and the reliability (calibration) of
such a probabilistic model. Hence, we introduce a cost function that encodes
their trade-off. We use the Continuous Rank Probability Score to measure
accuracy and we derive an analytic formula for the reliability, in the case of
forecasts of continuous scalar variables expressed in terms of Gaussian
distributions. The new Accuracy-Reliability cost function is then used to
estimate the input-dependent variance, given a black-box mean function, by
solving a two-objective optimization problem. The simple philosophy behind this
strategy is that predictions based on the estimated variances should not only
be accurate, but also reliable (i.e. statistical consistent with observations).
Conversely, early works based on the minimization of classical cost functions,
such as the negative log probability density, cannot simultaneously enforce
both accuracy and reliability. We show several examples both with synthetic
data, where the underlying hidden noise can accurately be recovered, and with
large real-world datasets.
