Revisiting SGD with Increasingly Weighted Averaging: Optimization and
  Generalization Perspectives
  Stochastic gradient descent (SGD) has been widely studied in the literature
from different angles, and is commonly employed for solving many big data
machine learning problems. However, the averaging technique, which combines all
iterative solutions into a single solution, is still under-explored. While some
increasingly weighted averaging schemes have been considered in the literature,
existing works are mostly restricted to strongly convex objective functions and
the convergence of optimization error. It remains unclear how these averaging
schemes affect the convergence of {\it both optimization error and
generalization error} (two equally important components of testing error) for
{\bf non-strongly convex objectives, including non-convex problems}. In this
paper, we {\it fill the gap} by comprehensively analyzing the increasingly
weighted averaging on convex, strongly convex and non-convex objective
functions in terms of both optimization error and generalization error. In
particular, we analyze a family of increasingly weighted averaging, where the
weight for the solution at iteration $t$ is proportional to $t^{\alpha}$
($\alpha > 0$). We show how $\alpha$ affects the optimization error and the
generalization error, and exhibit the trade-off caused by $\alpha$. Experiments
have demonstrated this trade-off and the effectiveness of polynomially
increased weighted averaging compared with other averaging schemes for a wide
range of problems including deep learning.
