On the expected behaviour of noise regularised deep neural networks as
  Gaussian processes
  Recent work has established the equivalence between deep neural networks and
Gaussian processes (GPs), resulting in so-called neural network Gaussian
processes (NNGPs). The behaviour of these models depends on the initialisation
of the corresponding network. In this work, we consider the impact of noise
regularisation (e.g. dropout) on NNGPs, and relate their behaviour to signal
propagation theory in noise regularised deep neural networks. For ReLU
activations, we find that the best performing NNGPs have kernel parameters that
correspond to a recently proposed initialisation scheme for noise regularised
ReLU networks. In addition, we show how the noise influences the covariance
matrix of the NNGP, producing a stronger prior towards simple functions away
from the training points. We verify our theoretical findings with experiments
on MNIST and CIFAR-10 as well as on synthetic data.
