Multiple Importance Sampling ELBO and Deep Ensembles of Variational
  Approximations
  In variational inference (VI), the marginal log-likelihood is estimated using
the standard evidence lower bound (ELBO), or improved versions as the
importance weighted ELBO (IWELBO). We propose the multiple importance sampling
ELBO (MISELBO), a \textit{versatile} yet \textit{simple} framework. MISELBO is
applicable in both amortized and classical VI, and it uses ensembles, e.g.,
deep ensembles, of independently inferred variational approximations. As far as
we are aware, the concept of deep ensembles in amortized VI has not previously
been established. We prove that MISELBO provides a tighter bound than the
average of standard ELBOs, and demonstrate empirically that it gives tighter
bounds than the average of IWELBOs. MISELBO is evaluated in density-estimation
experiments that include MNIST and several real-data phylogenetic tree
inference problems. First, on the MNIST dataset, MISELBO boosts the
density-estimation performances of a state-of-the-art model, nouveau VAE.
Second, in the phylogenetic tree inference setting, our framework enhances a
state-of-the-art VI algorithm that uses normalizing flows. On top of the
technical benefits of MISELBO, it allows to unveil connections between VI and
recent advances in the importance sampling literature, paving the way for
further methodological advances. We provide our code at
\url{https://github.com/Lagergren-Lab/MISELBO}.
