Stochastic Gradient Descent with Biased but Consistent Gradient
  Estimators
  Stochastic gradient descent (SGD), which dates back to the 1950s, is one of
the most popular and effective approaches for performing stochastic
optimization. Research on SGD resurged recently in machine learning for
optimizing convex loss functions and training nonconvex deep neural networks.
The theory assumes that one can easily compute an unbiased gradient estimator,
which is usually the case due to the sample average nature of empirical risk
minimization. There exist, however, many scenarios (e.g., graphs) where an
unbiased estimator may be as expensive to compute as the full gradient because
training examples are interconnected. Recently, Chen et al. (2018) proposed
using a consistent gradient estimator as an economic alternative. Encouraged by
empirical success, we show, in a general setting, that consistent estimators
result in the same convergence behavior as do unbiased ones. Our analysis
covers strongly convex, convex, and nonconvex objectives. We verify the results
with illustrative experiments on synthetic and real-world data. This work opens
several new research directions, including the development of more efficient
SGD updates with consistent estimators and the design of efficient training
algorithms for large-scale graphs.
