$\rho$-VAE: Autoregressive parametrization of the VAE encoder
  We make a minimal, but very effective alteration to the VAE model. This is
about a drop-in replacement for the (sample-dependent) approximate posterior to
change it from the standard white Gaussian with diagonal covariance to the
first-order autoregressive Gaussian. We argue that this is a more reasonable
choice to adopt for natural signals like images, as it does not force the
existing correlation in the data to disappear in the posterior. Moreover, it
allows more freedom for the approximate posterior to match the true posterior.
This allows for the repararametrization trick, as well as the KL-divergence
term to still have closed-form expressions, obviating the need for its
sample-based estimation. Although providing more freedom to adapt to correlated
distributions, our parametrization has even less number of parameters than the
diagonal covariance, as it requires only two scalars, $\rho$ and $s$, to
characterize correlation and scaling, respectively. As validated by the
experiments, our proposition noticeably and consistently improves the quality
of image generation in a plug-and-play manner, needing no further parameter
tuning, and across all setups. The code to reproduce our experiments is
available at \url{https://github.com/sssohrab/rho_VAE/}.
