Revisiting One-vs-All Classifiers for Predictive Uncertainty and
  Out-of-Distribution Detection in Neural Networks
  Accurate estimation of predictive uncertainty in modern neural networks is
critical to achieve well calibrated predictions and detect out-of-distribution
(OOD) inputs. The most promising approaches have been predominantly focused on
improving model uncertainty (e.g. deep ensembles and Bayesian neural networks)
and post-processing techniques for OOD detection (e.g. ODIN and Mahalanobis
distance). However, there has been relatively little investigation into how the
parametrization of the probabilities in discriminative classifiers affects the
uncertainty estimates, and the dominant method, softmax cross-entropy, results
in misleadingly high confidences on OOD data and under covariate shift. We
investigate alternative ways of formulating probabilities using (1) a
one-vs-all formulation to capture the notion of "none of the above", and (2) a
distance-based logit representation to encode uncertainty as a function of
distance to the training manifold. We show that one-vs-all formulations can
improve calibration on image classification tasks, while matching the
predictive performance of softmax without incurring any additional training or
test-time complexity.
