A Rule for Gradient Estimator Selection, with an Application to
  Variational Inference
  Stochastic gradient descent (SGD) is the workhorse of modern machine
learning. Sometimes, there are many different potential gradient estimators
that can be used. When so, choosing the one with the best tradeoff between cost
and variance is important. This paper analyzes the convergence rates of SGD as
a function of time, rather than iterations. This results in a simple rule to
select the estimator that leads to the best optimization convergence guarantee.
This choice is the same for different variants of SGD, and with different
assumptions about the objective (e.g. convexity or smoothness). Inspired by
this principle, we propose a technique to automatically select an estimator
when a finite pool of estimators is given. Then, we extend to infinite pools of
estimators, where each one is indexed by control variate weights. This is
enabled by a reduction to a mixed-integer quadratic program. Empirically,
automatically choosing an estimator performs comparably to the best estimator
chosen with hindsight.
