Probabilistic Neighbourhood Component Analysis: Sample Efficient
  Uncertainty Estimation in Deep Learning
  While Deep Neural Networks (DNNs) achieve state-of-the-art accuracy in
various applications, they often fall short in accurately estimating their
predictive uncertainty and, in turn, fail to recognize when these predictions
may be wrong. Several uncertainty-aware models, such as Bayesian Neural Network
(BNNs) and Deep Ensembles have been proposed in the literature for quantifying
predictive uncertainty. However, research in this area has been largely
confined to the big data regime. In this work, we show that the uncertainty
estimation capability of state-of-the-art BNNs and Deep Ensemble models
degrades significantly when the amount of training data is small. To address
the issue of accurate uncertainty estimation in the small-data regime, we
propose a probabilistic generalization of the popular sample-efficient
non-parametric kNN approach. Our approach enables deep kNN classifier to
accurately quantify underlying uncertainties in its prediction. We demonstrate
the usefulness of the proposed approach by achieving superior uncertainty
quantification as compared to state-of-the-art on a real-world application of
COVID-19 diagnosis from chest X-Rays. Our code is available at
https://github.com/ankurmallick/sample-efficient-uq
