Determinantal point processes based on orthogonal polynomials for
  sampling minibatches in SGD
  Stochastic gradient descent (SGD) is a cornerstone of machine learning. When
the number N of data items is large, SGD relies on constructing an unbiased
estimator of the gradient of the empirical risk using a small subset of the
original dataset, called a minibatch. Default minibatch construction involves
uniformly sampling a subset of the desired size, but alternatives have been
explored for variance reduction. In particular, experimental evidence suggests
drawing minibatches from determinantal point processes (DPPs), distributions
over minibatches that favour diversity among selected items. However, like in
recent work on DPPs for coresets, providing a systematic and principled
understanding of how and why DPPs help has been difficult. In this work, we
contribute an orthogonal polynomial-based DPP paradigm for minibatch sampling
in SGD. Our approach leverages the specific data distribution at hand, which
endows it with greater sensitivity and power over existing data-agnostic
methods. We substantiate our method via a detailed theoretical analysis of its
convergence properties, interweaving between the discrete data set and the
underlying continuous domain. In particular, we show how specific DPPs and a
string of controlled approximations can lead to gradient estimators with a
variance that decays faster with the batchsize than under uniform sampling.
Coupled with existing finite-time guarantees for SGD on convex objectives, this
entails that, DPP minibatches lead to a smaller bound on the mean square
approximation error than uniform minibatches. Moreover, our estimators are
amenable to a recent algorithm that directly samples linear statistics of DPPs
(i.e., the gradient estimator) without sampling the underlying DPP (i.e., the
minibatch), thereby reducing computational overhead. We provide detailed
synthetic as well as real data experiments to substantiate our theoretical
claims.
