Deep Neural Networks as Point Estimates for Deep Gaussian Processes
  Neural networks and Gaussian processes are complementary in their strengths
and weaknesses. Having a better understanding of their relationship comes with
the promise to make each method benefit from the strengths of the other. In
this work, we establish an equivalence between the forward passes of neural
networks and (deep) sparse Gaussian process models. The theory we develop is
based on interpreting activation functions as interdomain inducing features
through a rigorous analysis of the interplay between activation functions and
kernels. This results in models that can either be seen as neural networks with
improved uncertainty prediction or deep Gaussian processes with increased
prediction accuracy. These claims are supported by experimental results on
regression and classification datasets.
