The Implicit Regularization of Momentum Gradient Descent with Early
  Stopping
  The study on the implicit regularization induced by gradient-based
optimization is a longstanding pursuit. In the present paper, we characterize
the implicit regularization of momentum gradient descent (MGD) with early
stopping by comparing with the explicit $\ell_2$-regularization (ridge). In
details, we study MGD in the continuous-time view, so-called momentum gradient
flow (MGF), and show that its tendency is closer to ridge than the gradient
descent (GD) [Ali et al., 2019] for least squares regression. Moreover, we
prove that, under the calibration $t=\sqrt{2/\lambda}$, where $t$ is the time
parameter in MGF and $\lambda$ is the tuning parameter in ridge regression, the
risk of MGF is no more than 1.54 times that of ridge. In particular, the
relative Bayes risk of MGF to ridge is between 1 and 1.035 under the optimal
tuning. The numerical experiments support our theoretical results strongly.
