Uncertainty estimation under model misspecification in neural network
  regression
  Although neural networks are powerful function approximators, the underlying
modelling assumptions ultimately define the likelihood and thus the hypothesis
class they are parameterizing. In classification, these assumptions are minimal
as the commonly employed softmax is capable of representing any categorical
distribution. In regression, however, restrictive assumptions on the type of
continuous distribution to be realized are typically placed, like the dominant
choice of training via mean-squared error and its underlying Gaussianity
assumption. Recently, modelling advances allow to be agnostic to the type of
continuous distribution to be modelled, granting regression the flexibility of
classification models. While past studies stress the benefit of such flexible
regression models in terms of performance, here we study the effect of the
model choice on uncertainty estimation. We highlight that under model
misspecification, aleatoric uncertainty is not properly captured, and that a
Bayesian treatment of a misspecified model leads to unreliable epistemic
uncertainty estimates. Overall, our study provides an overview on how modelling
choices in regression may influence uncertainty estimation and thus any
downstream decision making process.
