Minimum Excess Risk in Bayesian Learning
  We analyze the best achievable performance of Bayesian learning under
generative models by defining and upper-bounding the minimum excess risk (MER):
the gap between the minimum expected loss attainable by learning from data and
the minimum expected loss that could be achieved if the model realization were
known. The definition of MER provides a principled way to define different
notions of uncertainties in Bayesian learning, including the aleatoric
uncertainty and the minimum epistemic uncertainty. Two methods for deriving
upper bounds for the MER are presented. The first method, generally suitable
for Bayesian learning with a parametric generative model, upper-bounds the MER
by the conditional mutual information between the model parameters and the
quantity being predicted given the observed data. It allows us to quantify the
rate at which the MER decays to zero as more data becomes available. Under
realizable models, this method also relates the MER to the richness of the
generative function class, notably the VC dimension in binary classification.
The second method, particularly suitable for Bayesian learning with a
parametric predictive model, relates the MER to the minimum estimation error of
the model parameters from data. It explicitly shows how the uncertainty in
model parameter estimation translates to the MER and to the final prediction
uncertainty. We also extend the definition and analysis of MER to the setting
with multiple model families and the setting with nonparametric models. Along
the discussions we draw some comparisons between the MER in Bayesian learning
and the excess risk in frequentist learning.
