Variance reduction properties of the reparameterization trick
  The reparameterization trick is widely used in variational inference as it
yields more accurate estimates of the gradient of the variational objective
than alternative approaches such as the score function method. Although there
is overwhelming empirical evidence in the literature showing its success, there
is relatively little research exploring why the reparameterization trick is so
effective. We explore this under the idealized assumptions that the variational
approximation is a mean-field Gaussian density and that the log of the joint
density of the model parameters and the data is a quadratic function that
depends on the variational mean. From this, we show that the marginal variances
of the reparameterization gradient estimator are smaller than those of the
score function gradient estimator. We apply the result of our idealized
analysis to real-world examples.
