Accuracy-Reliability Cost Function for Empirical Variance Estimation
  In this paper we focus on the problem of assigning uncertainties to
single-point predictions. We introduce a cost function that encodes the
trade-off between accuracy and reliability in probabilistic forecast. We derive
analytic formula for the case of forecasts of continuous scalar variables
expressed in terms of Gaussian distributions. The Accuracy-Reliability cost
function can be used to empirically estimate the variance in heteroskedastic
regression problems (input dependent noise), by solving a two-objective
optimization problem. The simple philosophy behind this strategy is that
predictions based on the estimated variances should be both accurate and
reliable (i.e. statistical consistent with observations). We show several
examples with synthetic data, where the underlying hidden noise function can be
accurately recovered, both in one and multi-dimensional problems. The practical
implementation of the method has been done using a Neural Network and, in the
one-dimensional case, with a simple polynomial fit.
