Training Learned Optimizers with Randomly Initialized Learned Optimizers
  Learned optimizers are increasingly effective, with performance exceeding
that of hand designed optimizers such as Adam~\citep{kingma2014adam} on
specific tasks \citep{metz2019understanding}. Despite the potential gains
available, in current work the meta-training (or `outer-training') of the
learned optimizer is performed by a hand-designed optimizer, or by an optimizer
trained by a hand-designed optimizer \citep{metz2020tasks}. We show that a
population of randomly initialized learned optimizers can be used to train
themselves from scratch in an online fashion, without resorting to a hand
designed optimizer in any part of the process. A form of population based
training is used to orchestrate this self-training. Although the randomly
initialized optimizers initially make slow progress, as they improve they
experience a positive feedback loop, and become rapidly more effective at
training themselves. We believe feedback loops of this type, where an optimizer
improves itself, will be important and powerful in the future of machine
learning. These methods not only provide a path towards increased performance,
but more importantly relieve research and engineering effort.
