Learning the Step-size Policy for the Limited-Memory
  Broyden-Fletcher-Goldfarb-Shanno Algorithm
  We consider the problem of how to learn a step-size policy for the
Limited-Memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm. This is a
limited computational memory quasi-Newton method widely used for deterministic
unconstrained optimization but currently avoided in large-scale problems for
requiring step sizes to be provided at each iteration. Existing methodologies
for the step size selection for L-BFGS use heuristic tuning of design
parameters and massive re-evaluations of the objective function and gradient to
find appropriate step-lengths. We propose a neural network architecture with
local information of the current iterate as the input. The step-length policy
is learned from data of similar optimization problems, avoids additional
evaluations of the objective function, and guarantees that the output step
remains inside a pre-defined interval. The corresponding training procedure is
formulated as a stochastic optimization problem using the backpropagation
through time algorithm. The performance of the proposed method is evaluated on
the training of classifiers for the MNIST database for handwritten digits and
for CIFAR-10. The results show that the proposed algorithm outperforms
heuristically tuned optimizers such as ADAM, RMSprop, L-BFGS with a
backtracking line search, and L-BFGS with a constant step size. The numerical
results also show that a learned policy can be used as a warm-start to train
new policies for different problems after a few additional training steps,
highlighting its potential use in multiple large-scale optimization problems.
