Amortized Proximal Optimization
  We propose a framework for online meta-optimization of parameters that govern
optimization, called Amortized Proximal Optimization (APO). We first interpret
various existing neural network optimizers as approximate stochastic proximal
point methods which trade off the current-batch loss with proximity terms in
both function space and weight space. The idea behind APO is to amortize the
minimization of the proximal point objective by meta-learning the parameters of
an update rule. We show how APO can be used to adapt a learning rate or a
structured preconditioning matrix. Under appropriate assumptions, APO can
recover existing optimizers such as natural gradient descent and KFAC. It
enjoys low computational overhead and avoids expensive and numerically
sensitive operations required by some second-order optimizers, such as matrix
inverses. We empirically test APO for online adaptation of learning rates and
structured preconditioning matrices for regression, image reconstruction, image
classification, and natural language translation tasks. Empirically, the
learning rate schedules found by APO generally outperform optimal fixed
learning rates and are competitive with manually tuned decay schedules. Using
APO to adapt a structured preconditioning matrix generally results in
optimization performance competitive with second-order methods. Moreover, the
absence of matrix inversion provides numerical stability, making it effective
for low precision training.
