Optimal Posteriors for Chi-squared Divergence based PAC-Bayesian Bounds
  and Comparison with KL-divergence based Optimal Posteriors and
  Cross-Validation Procedure
  We investigate optimal posteriors for recently introduced \cite{begin2016pac}
chi-squared divergence based PAC-Bayesian bounds in terms of nature of their
distribution, scalability of computations, and test set performance. For a
finite classifier set, we deduce bounds for three distance functions:
KL-divergence, linear and squared distances. Optimal posterior weights are
proportional to deviations of empirical risks, usually with subset support. For
uniform prior, it is sufficient to search among posteriors on classifier
subsets ordered by these risks. We show the bound minimization for linear
distance as a convex program and obtain a closed-form expression for its
optimal posterior. Whereas that for squared distance is a quasi-convex program
under a specific condition, and the one for KL-divergence is non-convex
optimization (a difference of convex functions). To compute such optimal
posteriors, we derive fast converging fixed point (FP) equations. We apply
these approaches to a finite set of SVM regularization parameter values to
yield stochastic SVMs with tight bounds. We perform a comprehensive performance
comparison between our optimal posteriors and known KL-divergence based
posteriors on a variety of UCI datasets with varying ranges and variances in
risk values, etc. Chi-squared divergence based posteriors have weaker bounds
and worse test errors, hinting at an underlying regularization by KL-divergence
based posteriors. Our study highlights the impact of divergence function on the
performance of PAC-Bayesian classifiers. We compare our stochastic classifiers
with cross-validation based deterministic classifier. The latter has better
test errors, but ours is more sample robust, has quantifiable generalization
guarantees, and is computationally much faster.
