A Simple Guard for Learned Optimizers
  If the trend of learned components eventually outperforming their
hand-crafted version continues, learned optimizers will eventually outperform
hand-crafted optimizers like SGD or Adam. Even if learned optimizers (L2Os)
eventually outpace hand-crafted ones in practice however, they are still not
provably convergent and might fail out of distribution. These are the questions
addressed here. Currently, learned optimizers frequently outperform generic
hand-crafted optimizers (such as gradient descent) at the beginning of learning
but they generally plateau after some time while the generic algorithms
continue to make progress and often overtake the learned algorithm as Aesop's
tortoise which overtakes the hare and are not. L2Os also still have a difficult
time generalizing out of distribution. (Heaton et al., 2020) proposed
Safeguarded L2O (GL2O) which can take a learned optimizer and safeguard it with
a generic learning algorithm so that by conditionally switching between the
two, the resulting algorithm is provably convergent.
  We propose a new class of Safeguarded L2O, called Loss-Guarded L2O (LGL2O),
which is both conceptually simpler and computationally less expensive. The
guarding mechanism decides solely based on the expected future loss value of
both optimizers. Furthermore, we show theoretical proof of LGL2O's convergence
guarantee and empirical results comparing to GL2O and other baselines showing
that it combines the best of both L2O and SGD and and in practice converges
much better than GL2O.
