Interpretable deep Gaussian processes with moments
  Deep Gaussian Processes (DGPs) combine the expressiveness of Deep Neural
Networks (DNNs) with quantified uncertainty of Gaussian Processes (GPs).
Expressive power and intractable inference both result from the non-Gaussian
distribution over composition functions. We propose interpretable DGP based on
approximating DGP as a GP by calculating the exact moments, which additionally
identify the heavy-tailed nature of some DGP distributions. Consequently, our
approach admits interpretation as both NNs with specified activation functions
and as a variational approximation to DGP. We identify the expressivity
parameter of DGP and find non-local and non-stationary correlation from DGP
composition. We provide general recipes for deriving the effective kernels for
DGP of two, three, or infinitely many layers, composed of homogeneous or
heterogeneous kernels. Results illustrate the expressiveness of our effective
kernels through samples from the prior and inference on simulated and real data
and demonstrate advantages of interpretability by analysis of analytic forms,
and draw relations and equivalences across kernels.
