A Generalization Bound for Online Variational Inference
  Bayesian inference provides an attractive online-learning framework to
analyze sequential data, and offers generalization guarantees which hold even
with model mismatch and adversaries. Unfortunately, exact Bayesian inference is
rarely feasible in practice and approximation methods are usually employed, but
do such methods preserve the generalization properties of Bayesian inference ?
In this paper, we show that this is indeed the case for some variational
inference (VI) algorithms. We consider a few existing online, tempered VI
algorithms, as well as a new algorithm, and derive their generalization bounds.
Our theoretical result relies on the convexity of the variational objective,
but we argue that the result should hold more generally and present empirical
evidence in support of this. Our work in this paper presents theoretical
justifications in favor of online algorithms relying on approximate Bayesian
methods.
