Predictive Uncertainty Quantification with Compound Density Networks
  Despite the huge success of deep neural networks (NNs), finding good
mechanisms for quantifying their prediction uncertainty is still an open
problem. Bayesian neural networks are one of the most popular approaches to
uncertainty quantification. On the other hand, it was recently shown that
ensembles of NNs, which belong to the class of mixture models, can be used to
quantify prediction uncertainty. In this paper, we build upon these two
approaches. First, we increase the mixture model's flexibility by replacing the
fixed mixing weights by an adaptive, input-dependent distribution (specifying
the probability of each component) represented by NNs, and by considering
uncountably many mixture components. The resulting class of models can be seen
as the continuous counterpart to mixture density networks and is therefore
referred to as compound density networks (CDNs). We employ both maximum
likelihood and variational Bayesian inference to train CDNs, and empirically
show that they yield better uncertainty estimates on out-of-distribution data
and are more robust to adversarial examples than the previous approaches.
