A Sparse Expansion For Deep Gaussian Processes
  Deep Gaussian Processes (DGP) enable a non-parametric approach to quantify
the uncertainty of complex deep machine learning models. Conventional
inferential methods for DGP models can suffer from high computational
complexity as they require large-scale operations with kernel matrices for
training and inference. In this work, we propose an efficient scheme for
accurate inference and prediction based on a range of Gaussian Processes,
called the Tensor Markov Gaussian Processes (TMGP). We construct an induced
approximation of TMGP referred to as the hierarchical expansion. Next, we
develop a deep TMGP (DTMGP) model as the composition of multiple hierarchical
expansion of TMGPs. The proposed DTMGP model has the following properties: (1)
the outputs of each activation function are deterministic while the weights are
chosen independently from standard Gaussian distribution; (2) in training or
prediction, only O(polylog(M)) (out of M) activation functions have non-zero
outputs, which significantly boosts the computational efficiency. Our numerical
experiments on real datasets show the superior computational efficiency of
DTMGP versus other DGP models.
