A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement
  Neural networks are ubiquitous in many tasks, but trusting their predictions
is an open issue. Uncertainty quantification is required for many applications,
and disentangled aleatoric and epistemic uncertainties are best. In this paper,
we generalize methods to produce disentangled uncertainties to work with
different uncertainty quantification methods, and evaluate their capability to
produce disentangled uncertainties. Our results show that: there is an
interaction between learning aleatoric and epistemic uncertainty, which is
unexpected and violates assumptions on aleatoric uncertainty, some methods like
Flipout produce zero epistemic uncertainty, aleatoric uncertainty is unreliable
in the out-of-distribution setting, and Ensembles provide overall the best
disentangling quality. We also explore the error produced by the number of
samples hyper-parameter in the sampling softmax function, recommending N > 100
samples. We expect that our formulation and results help practitioners and
researchers choose uncertainty methods and expand the use of disentangled
uncertainties, as well as motivate additional research into this topic.
