Practical tradeoffs between memory, compute, and performance in learned
  optimizers
  Optimization plays a costly and crucial role in developing machine learning
systems. In learned optimizers, the few hyperparameters of commonly used
hand-designed optimizers, e.g. Adam or SGD, are replaced with flexible
parametric functions. The parameters of these functions are then optimized so
that the resulting learned optimizer minimizes a target loss on a chosen class
of models. Learned optimizers can both reduce the number of required training
steps and improve the final test loss. However, they can be expensive to train,
and once trained can be expensive to use due to computational and memory
overhead for the optimizer itself. In this work, we identify and quantify the
design features governing the memory, compute, and performance trade-offs for
many learned and hand-designed optimizers. We further leverage our analysis to
construct a learned optimizer that is both faster and more memory efficient
than previous work.
