Theoretical characterization of uncertainty in high-dimensional linear
  classification
  Being able to reliably assess not only the accuracy but also the uncertainty
of models' predictions is an important endeavour in modern machine learning.
Even if the model generating the data and labels is known, computing the
intrinsic uncertainty after learning the model from a limited number of samples
amounts to sampling the corresponding posterior probability measure. Such
sampling is computationally challenging in high-dimensional problems and
theoretical results on heuristic uncertainty estimators in high-dimensions are
thus scarce. In this manuscript, we characterise uncertainty for learning from
limited number of samples of high-dimensional Gaussian input data and labels
generated by the probit model. We prove that the Bayesian uncertainty (i.e. the
posterior marginals) can be asymptotically obtained by the approximate message
passing algorithm, bypassing the canonical but costly Monte Carlo sampling of
the posterior. We then provide a closed-form formula for the joint statistics
between the logistic classifier, the uncertainty of the statistically optimal
Bayesian classifier and the ground-truth probit uncertainty. The formula allows
us to investigate calibration of the logistic classifier learning from limited
amount of samples. We discuss how over-confidence can be mitigated by
appropriately regularising, and show that cross-validating with respect to the
loss leads to better calibration than with the 0/1 error.
