Reliable training and estimation of variance networks
  We propose and investigate new complementary methodologies for estimating
predictive variance networks in regression neural networks. We derive a locally
aware mini-batching scheme that result in sparse robust gradients, and show how
to make unbiased weight updates to a variance network. Further, we formulate a
heuristic for robustly fitting both the mean and variance networks post hoc.
Finally, we take inspiration from posterior Gaussian processes and propose a
network architecture with similar extrapolation properties to Gaussian
processes. The proposed methodologies are complementary, and improve upon
baseline methods individually. Experimentally, we investigate the impact on
predictive uncertainty on multiple datasets and tasks ranging from regression,
active learning and generative modeling. Experiments consistently show
significant improvements in predictive uncertainty estimation over
state-of-the-art methods across tasks and datasets.
