Characteristics of Monte Carlo Dropout in Wide Neural Networks
  Monte Carlo (MC) dropout is one of the state-of-the-art approaches for
uncertainty estimation in neural networks (NNs). It has been interpreted as
approximately performing Bayesian inference. Based on previous work on the
approximation of Gaussian processes by wide and deep neural networks with
random weights, we study the limiting distribution of wide untrained NNs under
dropout more rigorously and prove that they as well converge to Gaussian
processes for fixed sets of weights and biases. We sketch an argument that this
property might also hold for infinitely wide feed-forward networks that are
trained with (full-batch) gradient descent. The theory is contrasted by an
empirical analysis in which we find correlations and non-Gaussian behaviour for
the pre-activations of finite width NNs. We therefore investigate how
(strongly) correlated pre-activations can induce non-Gaussian behavior in NNs
with strongly correlated weights.
