Learning an Adaptive Learning Rate Schedule
  The learning rate is one of the most important hyper-parameters for model
training and generalization. However, current hand-designed parametric learning
rate schedules offer limited flexibility and the predefined schedule may not
match the training dynamics of high dimensional and non-convex optimization
problems. In this paper, we propose a reinforcement learning based framework
that can automatically learn an adaptive learning rate schedule by leveraging
the information from past training histories. The learning rate dynamically
changes based on the current training dynamics. To validate this framework, we
conduct experiments with different neural network architectures on the Fashion
MINIST and CIFAR10 datasets. Experimental results show that the auto-learned
learning rate controller can achieve better test results. In addition, the
trained controller network is generalizable -- able to be trained on one data
set and transferred to new problems.
