The Statistical Complexity of Early-Stopped Mirror Descent
  Recently there has been a surge of interest in understanding implicit
regularization properties of iterative gradient-based optimization algorithms.
In this paper, we study the statistical guarantees on the excess risk achieved
by early-stopped unconstrained mirror descent algorithms applied to the
unregularized empirical risk with the squared loss for linear models and kernel
methods. By completing an inequality that characterizes convexity for the
squared loss, we identify an intrinsic link between offset Rademacher
complexities and potential-based convergence analysis of mirror descent
methods. Our observation immediately yields excess risk guarantees for the path
traced by the iterates of mirror descent in terms of offset complexities of
certain function classes depending only on the choice of the mirror map,
initialization point, step-size, and the number of iterations. We apply our
theory to recover, in a clean and elegant manner via rather short proofs, some
of the recent results in the implicit regularization literature, while also
showing how to improve upon them in some settings.
