A Stochastic Gradient Method with Biased Estimation for Faster Nonconvex
  Optimization
  A number of optimization approaches have been proposed for optimizing
nonconvex objectives (e.g. deep learning models), such as batch gradient
descent, stochastic gradient descent and stochastic variance reduced gradient
descent. Theory shows these optimization methods can converge by using an
unbiased gradient estimator. However, in practice biased gradient estimation
can allow more efficient convergence to the vicinity since an unbiased approach
is computationally more expensive. To produce fast convergence there are two
trade-offs of these optimization strategies which are between stochastic/batch,
and between biased/unbiased. This paper proposes an integrated approach which
can control the nature of the stochastic element in the optimizer and can
balance the trade-off of estimator between the biased and unbiased by using a
hyper-parameter. It is shown theoretically and experimentally that this
hyper-parameter can be configured to provide an effective balance to improve
the convergence rate.
