n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive
  Uncertainty and True Error
  As deep learning applications are becoming more and more pervasive in
robotics, the question of evaluating the reliability of inferences becomes a
central question in the robotics community. This domain, known as predictive
uncertainty, has come under the scrutiny of research groups developing Bayesian
approaches adapted to deep learning such as Monte Carlo Dropout. Unfortunately,
for the time being, the real goal of predictive uncertainty has been swept
under the rug. Indeed, these approaches are solely evaluated in terms of raw
performance of the network prediction, while the quality of their estimated
uncertainty is not assessed. Evaluating such uncertainty prediction quality is
especially important in robotics, as actions shall depend on the confidence in
perceived information. In this context, the main contribution of this article
is to propose a novel metric that is adapted to the evaluation of relative
uncertainty assessment and directly applicable to regression with deep neural
networks. To experimentally validate this metric, we evaluate it on a toy
dataset and then apply it to the task of monocular depth estimation.
