Ultra-fast Deep Mixtures of Gaussian Process Experts
  Mixtures of experts have become an indispensable tool for flexible modelling
in a supervised learning context, and sparse Gaussian processes (GP) have shown
promise as a leading candidate for the experts in such models. In this article,
we propose to design the gating network for selecting the experts from such
mixtures of sparse GPs using a deep neural network (DNN). Furthermore, an
ultra-fast one pass algorithm called Cluster-Classify-Regress (CCR) is
leveraged to approximate the maximum a posteriori (MAP) estimator extremely
quickly. This powerful combination of model and algorithm together delivers a
novel method which is flexible, robust, and extremely efficient. In particular,
the method is able to outperform competing methods in terms of accuracy and
uncertainty quantification. The cost is competitive on low-dimensional and
small data sets but is significantly lower for higher dimensional and big data
sets. Iteratively maximizing the distribution of experts given allocations and
allocations given experts does not provide significant improvement, which
indicates that the algorithm achieves a good approximation to the local MAP
estimator very fast. This insight can be useful also in the context of other
mixture of experts models.
