REX: Revisiting Budgeted Training with an Improved Schedule
  Deep learning practitioners often operate on a computational and monetary
budget. Thus, it is critical to design optimization algorithms that perform
well under any budget. The linear learning rate schedule is considered the best
budget-aware schedule, as it outperforms most other schedules in the low budget
regime. On the other hand, learning rate schedules -- such as the
\texttt{30-60-90} step schedule -- are known to achieve high performance when
the model can be trained for many epochs. Yet, it is often not known a priori
whether one's budget will be large or small; thus, the optimal choice of
learning rate schedule is made on a case-by-case basis. In this paper, we frame
the learning rate schedule selection problem as a combination of $i)$ selecting
a profile (i.e., the continuous function that models the learning rate
schedule), and $ii)$ choosing a sampling rate (i.e., how frequently the
learning rate is updated/sampled from this profile). We propose a novel profile
and sampling rate combination called the Reflected Exponential (REX) schedule,
which we evaluate across seven different experimental settings with both SGD
and Adam optimizers. REX outperforms the linear schedule in the low budget
regime, while matching or exceeding the performance of several state-of-the-art
learning rate schedules (linear, step, exponential, cosine, step decay on
plateau, and OneCycle) in both high and low budget regimes. Furthermore, REX
requires no added computation, storage, or hyperparameters.
