Scaling transition from momentum stochastic gradient descent to plain
  stochastic gradient descent
  The plain stochastic gradient descent and momentum stochastic gradient
descent have extremely wide applications in deep learning due to their simple
settings and low computational complexity. The momentum stochastic gradient
descent uses the accumulated gradient as the updated direction of the current
parameters, which has a faster training speed. Because the direction of the
plain stochastic gradient descent has not been corrected by the accumulated
gradient. For the parameters that currently need to be updated, it is the
optimal direction, and its update is more accurate. We combine the advantages
of the momentum stochastic gradient descent with fast training speed and the
plain stochastic gradient descent with high accuracy, and propose a scaling
transition from momentum stochastic gradient descent to plain stochastic
gradient descent(TSGD) method. At the same time, a learning rate that decreases
linearly with the iterations is used instead of a constant learning rate. The
TSGD algorithm has a larger step size in the early stage to speed up the
training, and training with a smaller step size in the later stage can steadily
converge. Our experimental results show that the TSGD algorithm has faster
training speed, higher accuracy and better stability. Our implementation is
available at: https://github.com/kunzeng/TSGD.
