Practical calibration of the temperature parameter in Gibbs posteriors
  PAC-Bayesian algorithms and Gibbs posteriors are gaining popularity due to
their robustness against model misspecification even when Bayesian inference is
inconsistent. The PAC-Bayesian alpha-posterior is a generalization of the
standard Bayes posterior which can be tempered with a parameter alpha to handle
inconsistency. Data driven methods for tuning alpha have been proposed but are
still few, and are often computationally heavy. Additionally, the adequacy of
these methods in cases where we use variational approximations instead of exact
alpha-posteriors is not clear. This narrows their usage to simple models and
prevents their application to large-scale problems. We hence need fast methods
to tune alpha that work with both exact and variational alpha-posteriors.
First, we propose two data driven methods for tuning alpha, based on
sample-splitting and bootstrapping respectively. Second, we formulate the
(exact or variational) posteriors of three popular statistical models, and
modify them into alpha-posteriors. For each model, we test our strategies and
compare them with standard Bayes and Grunwald's SafeBayes. While bootstrapping
achieves mixed results, sample-splitting and SafeBayes perform well on the
exact and variational alpha-posteriors we describe, and achieve better results
than standard Bayes in misspecified or complex models. Additionally,
sample-splitting outperforms SafeBayes in terms of speed. Sample-splitting
offers a fast and easy solution to inconsistency and typically performs
similarly or better than Bayesian inference. Our results provide hints on the
calibration of alpha in PAC-Bayesian and Gibbs posteriors, and may facilitate
using these methods in large and complex models.
