Robust PAC$^m$: Training Ensemble Models Under Model Misspecification
  and Outliers
  Standard Bayesian learning is known to have suboptimal generalization
capabilities under model misspecification and in the presence of outliers.
PAC-Bayes theory demonstrates that the free energy criterion minimized by
Bayesian learning is a bound on the generalization error for Gibbs predictors
(i.e., for single models drawn at random from the posterior) under the
assumption of sampling distributions uncontaminated by outliers. This viewpoint
provides a justification for the limitations of Bayesian learning when the
model is misspecified, requiring ensembling, and when data is affected by
outliers. In recent work, PAC-Bayes bounds - referred to as PAC$^m$ - were
derived to introduce free energy metrics that account for the performance of
ensemble predictors, obtaining enhanced performance under misspecification.
This work presents a novel robust free energy criterion that combines the
generalized logarithm score function with PAC$^m$ ensemble bounds. The proposed
free energy training criterion produces predictive distributions that are able
to concurrently counteract the detrimental effects of model misspecification
and outliers.
