Stagewise Training Accelerates Convergence of Testing Error Over SGD
  Stagewise training strategy is widely used for learning neural networks,
which runs a stochastic algorithm (e.g., SGD) starting with a relatively large
step size (aka learning rate) and geometrically decreasing the step size after
a number of iterations. It has been observed that the stagewise SGD has much
faster convergence than the vanilla SGD with a polynomially decaying step size
in terms of both training error and testing error. {\it But how to explain this
phenomenon has been largely ignored by existing studies.} This paper provides
some theoretical evidence for explaining this faster convergence. In
particular, we consider a stagewise training strategy for minimizing empirical
risk that satisfies the Polyak-\L ojasiewicz (PL) condition, which has been
observed/proved for neural networks and also holds for a broad family of convex
functions. For convex loss functions and two classes of "nice-behaviored"
non-convex objectives that are close to a convex function, we establish faster
convergence of stagewise training than the vanilla SGD under the PL condition
on both training error and testing error. Experiments on stagewise learning of
deep residual networks exhibits that it satisfies one type of non-convexity
assumption and therefore can be explained by our theory. Of independent
interest, the testing error bounds for the considered non-convex loss functions
are dimensionality and norm independent.
