On Connecting Deep Trigonometric Networks with Deep Gaussian Processes:
  Covariance, Expressivity, and Neural Tangent Kernel
  Deep Gaussian Process (DGP) as a model prior in Bayesian learning intuitively
exploits the expressive power in function composition. DGPs also offer diverse
modeling capabilities, but inference becomes Achilles' heel as marginalization
in latent function space is not tractable. With Bochner's theorem, DGP with
squared exponential kernel can be viewed as a deep trigonometric network
consisting of the random feature layers, sine and cosine activation units, and
random weight layers. In the wide limit with a bottleneck, we show that the
weight space view yield the same effective covariance functions which were
obtained previously in function space. As such, DGPs can be translated into the
deep trig networks, which is flexible and expressive as one can freely adopt
different prior distributions over the parameters.Interestingly, the network
representation enables the study of DGP's neural tangent kernel, which may
reveal the mean of the intractable predictive distribution. Statistically,
unlike the shallow networks, deep networks of finite width have covariance
deviating from the limiting kernel, and the inner and outer widths may play
different roles in learning.
