SMG: A Shuffling Gradient-Based Method with Momentum
  We combine two advanced ideas widely used in optimization for machine
learning: shuffling strategy and momentum technique to develop a novel
shuffling gradient-based method with momentum, coined Shuffling Momentum
Gradient (SMG), for non-convex finite-sum optimization problems. While our
method is inspired by momentum techniques, its update is fundamentally
different from existing momentum-based methods. We establish state-of-the-art
convergence rates of SMG for any shuffling strategy using either constant or
diminishing learning rate under standard assumptions (i.e.$L$-smoothness and
bounded variance). When the shuffling strategy is fixed, we develop another new
algorithm that is similar to existing momentum methods, and prove the same
convergence rates for this algorithm under the $L$-smoothness and bounded
gradient assumptions. We demonstrate our algorithms via numerical simulations
on standard datasets and compare them with existing shuffling methods. Our
tests have shown encouraging performance of the new algorithms.
