Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo
  Objectives
  We revisit the theory of importance weighted variational inference (IWVI), a
promising strategy for learning latent variable models. IWVI uses new
variational bounds, known as Monte Carlo objectives (MCOs), obtained by
replacing intractable integrals by Monte Carlo estimates -- usually simply
obtained via importance sampling. Burda, Grosse and Salakhutdinov (2016) showed
that increasing the number of importance samples provably tightens the gap
between the bound and the likelihood. Inspired by this simple monotonicity
theorem, we present a series of nonasymptotic results that link properties of
Monte Carlo estimates to tightness of MCOs. We challenge the rationale that
smaller Monte Carlo variance leads to better bounds. We confirm theoretically
the empirical findings of several recent papers by showing that, in a precise
sense, negative correlation reduces the variational gap. We also generalise the
original monotonicity theorem by considering non-uniform weights. We discuss
several practical consequences of our theoretical results. Our work borrows
many ideas and results from the theory of stochastic orders.
