Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise
  Variance Parameterization
  Brittle optimization has been observed to adversely impact model likelihoods
for regression and VAEs when simultaneously fitting neural network mappings
from a (random) variable onto the mean and variance of a dependent Gaussian
variable. Previous works have bolstered optimization and improved likelihoods,
but fail other basic posterior predictive checks (PPCs). Under the PPC
framework, we propose critiques to test predictive mean and variance
calibration and the predictive distribution's ability to generate sensible
data. We find that our attractively simple solution, to treat heteroscedastic
variance variationally, sufficiently regularizes variance to pass these PPCs.
We consider a diverse gamut of existing and novel priors and find our methods
preserve or outperform existing model likelihoods while significantly improving
parameter calibration and sample quality for regression and VAEs.
