Universal Boosting Variational Inference
  Boosting variational inference (BVI) approximates an intractable probability
density by iteratively building up a mixture of simple component distributions
one at a time, using techniques from sparse convex optimization to provide both
computational scalability and approximation error guarantees. But the
guarantees have strong conditions that do not often hold in practice, resulting
in degenerate component optimization problems; and we show that the ad-hoc
regularization used to prevent degeneracy in practice can cause BVI to fail in
unintuitive ways. We thus develop universal boosting variational inference
(UBVI), a BVI scheme that exploits the simple geometry of probability densities
under the Hellinger metric to prevent the degeneracy of other gradient-based
BVI methods, avoid difficult joint optimizations of both component and weight,
and simplify fully-corrective weight optimizations. We show that for any target
density and any mixture component family, the output of UBVI converges to the
best possible approximation in the mixture family, even when the mixture family
is misspecified. We develop a scalable implementation based on exponential
family mixture components and standard stochastic optimization techniques.
Finally, we discuss statistical benefits of the Hellinger distance as a
variational objective through bounds on posterior probability, moment, and
importance sampling errors. Experiments on multiple datasets and models show
that UBVI provides reliable, accurate posterior approximations.
