On the adequacy of untuned warmup for adaptive optimization
  Adaptive optimization algorithms such as Adam are widely used in deep
learning. The stability of such algorithms is often improved with a warmup
schedule for the learning rate. Motivated by the difficulty of choosing and
tuning warmup schedules, recent work proposes automatic variance rectification
of Adam's adaptive learning rate, claiming that this rectified approach
("RAdam") surpasses the vanilla Adam algorithm and reduces the need for
expensive tuning of Adam with warmup. In this work, we refute this analysis and
provide an alternative explanation for the necessity of warmup based on the
magnitude of the update term, which is of greater relevance to training
stability. We then provide some "rule-of-thumb" warmup schedules, and we
demonstrate that simple untuned warmup of Adam performs more-or-less
identically to RAdam in typical practical settings. We conclude by suggesting
that practitioners stick to linear warmup with Adam, with a sensible default
being linear warmup over $2 / (1 - \beta_2)$ training iterations.
