Variational Inference with Holder Bounds
  The recent introduction of thermodynamic integration techniques has provided
a new framework for understanding and improving variational inference (VI). In
this work, we present a careful analysis of the thermodynamic variational
objective (TVO), bridging the gap between existing variational objectives and
shedding new insights to advance the field. In particular, we elucidate how the
TVO naturally connects the three key variational schemes, namely the
importance-weighted VI, Renyi-VI, and MCMC-VI, which subsumes most VI
objectives employed in practice. To explain the performance gap between theory
and practice, we reveal how the pathological geometry of thermodynamic curves
negatively affects TVO. By generalizing the integration path from the geometric
mean to the weighted Holder mean, we extend the theory of TVO and identify new
opportunities for improving VI. This motivates our new VI objectives, named the
Holder bounds, which flatten the thermodynamic curves and promise to achieve a
one-step approximation of the exact marginal log-likelihood. A comprehensive
discussion on the choices of numerical estimators is provided. We present
strong empirical evidence on both synthetic and real-world datasets to support
our claims.
