Boosting Variational Inference With Locally Adaptive Step-Sizes
  Variational Inference makes a trade-off between the capacity of the
variational family and the tractability of finding an approximate posterior
distribution. Instead, Boosting Variational Inference allows practitioners to
obtain increasingly good posterior approximations by spending more compute. The
main obstacle to widespread adoption of Boosting Variational Inference is the
amount of resources necessary to improve over a strong Variational Inference
baseline. In our work, we trace this limitation back to the global curvature of
the KL-divergence. We characterize how the global curvature impacts time and
memory consumption, address the problem with the notion of local curvature, and
provide a novel approximate backtracking algorithm for estimating local
curvature. We give new theoretical convergence rates for our algorithms and
provide experimental validation on synthetic and real-world datasets.
