First-Order Adaptive Sample Size Methods to Reduce Complexity of
  Empirical Risk Minimization
  This paper studies empirical risk minimization (ERM) problems for large-scale
datasets and incorporates the idea of adaptive sample size methods to improve
the guaranteed convergence bounds for first-order stochastic and deterministic
methods. In contrast to traditional methods that attempt to solve the ERM
problem corresponding to the full dataset directly, adaptive sample size
schemes start with a small number of samples and solve the corresponding ERM
problem to its statistical accuracy. The sample size is then grown
geometrically -- e.g., scaling by a factor of two -- and use the solution of
the previous ERM as a warm start for the new ERM. Theoretical analyses show
that the use of adaptive sample size methods reduces the overall computational
cost of achieving the statistical accuracy of the whole dataset for a broad
range of deterministic and stochastic first-order methods. The gains are
specific to the choice of method. When particularized to, e.g., accelerated
gradient descent and stochastic variance reduce gradient, the computational
cost advantage is a logarithm of the number of training samples. Numerical
experiments on various datasets confirm theoretical claims and showcase the
gains of using the proposed adaptive sample size scheme.
