How Tight Can PAC-Bayes be in the Small Data Regime?
  In this paper, we investigate the question: Given a small number of
datapoints, for example N = 30, how tight can PAC-Bayes and test set bounds be
made? For such small datasets, test set bounds adversely affect generalisation
performance by withholding data from the training procedure. In this setting,
PAC-Bayes bounds are especially attractive, due to their ability to use all the
data to simultaneously learn a posterior and bound its generalisation risk. We
focus on the case of i.i.d. data with a bounded loss and consider the generic
PAC-Bayes theorem of Germain et al. While their theorem is known to recover
many existing PAC-Bayes bounds, it is unclear what the tightest bound derivable
from their framework is. For a fixed learning algorithm and dataset, we show
that the tightest possible bound coincides with a bound considered by Catoni;
and, in the more natural case of distributions over datasets, we establish a
lower bound on the best bound achievable in expectation. Interestingly, this
lower bound recovers the Chernoff test set bound if the posterior is equal to
the prior. Moreover, to illustrate how tight these bounds can be, we study
synthetic one-dimensional classification tasks in which it is feasible to
meta-learn both the prior and the form of the bound to numerically optimise for
the tightest bounds possible. We find that in this simple, controlled scenario,
PAC-Bayes bounds are competitive with comparable, commonly used Chernoff test
set bounds. However, the sharpest test set bounds still lead to better
guarantees on the generalisation error than the PAC-Bayes bounds we consider.
