AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on
  the Fly
  The learning rate (LR) schedule is one of the most important hyper-parameters
needing careful tuning in training DNNs. However, it is also one of the least
automated parts of machine learning systems and usually costs significant
manual effort and computing. Though there are pre-defined LR schedules and
optimizers with adaptive LR, they introduce new hyperparameters that need to be
tuned separately for different tasks/datasets. In this paper, we consider the
question: Can we automatically tune the LR over the course of training without
human involvement? We propose an efficient method, AutoLRS, which automatically
optimizes the LR for each training stage by modeling training dynamics. AutoLRS
aims to find an LR applied to every $\tau$ steps that minimizes the resulted
validation loss. We solve this black-box optimization on the fly by Bayesian
optimization (BO). However, collecting training instances for BO requires a
system to evaluate each LR queried by BO's acquisition function for $\tau$
steps, which is prohibitively expensive in practice. Instead, we apply each
candidate LR for only $\tau'\ll\tau$ steps and train an exponential model to
predict the validation loss after $\tau$ steps. This mutual-training process
between BO and the loss-prediction model allows us to limit the training steps
invested in the BO search. We demonstrate the advantages and the generality of
AutoLRS through extensive experiments of training DNNs for tasks from diverse
domains using different optimizers. The LR schedules auto-generated by AutoLRS
lead to a speedup of $1.22\times$, $1.43\times$, and $1.5\times$ when training
ResNet-50, Transformer, and BERT, respectively, compared to the LR schedules in
their original papers, and an average speedup of $1.31\times$ over
state-of-the-art heavily-tuned LR schedules.
