Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive
  Sample Size Approach
  In this paper, we study the application of quasi-Newton methods for solving
empirical risk minimization (ERM) problems defined over a large dataset.
Traditional deterministic and stochastic quasi-Newton methods can be executed
to solve such problems; however, it is known that their global convergence rate
may not be better than first-order methods, and their local superlinear
convergence only appears towards the end of the learning process. In this
paper, we use an adaptive sample size scheme that exploits the superlinear
convergence of quasi-Newton methods globally and throughout the entire learning
process. The main idea of the proposed adaptive sample size algorithms is to
start with a small subset of data points and solve their corresponding ERM
problem within its statistical accuracy, and then enlarge the sample size
geometrically and use the optimal solution of the problem corresponding to the
smaller set as an initial point for solving the subsequent ERM problem with
more samples. We show that if the initial sample size is sufficiently large and
we use quasi-Newton methods to solve each subproblem, the subproblems can be
solved superlinearly fast (after at most three iterations), as we guarantee
that the iterates always stay within a neighborhood that quasi-Newton methods
converge superlinearly. Numerical experiments on various datasets confirm our
theoretical results and demonstrate the computational advantages of our method.
