Learning Gradient Descent: Better Generalization and Longer Horizons
  Training deep neural networks is a highly nontrivial task, involving
carefully selecting appropriate training algorithms, scheduling step sizes and
tuning other hyperparameters. Trying different combinations can be quite
labor-intensive and time consuming. Recently, researchers have tried to use
deep learning algorithms to exploit the landscape of the loss function of the
training problem of interest, and learn how to optimize over it in an automatic
way. In this paper, we propose a new learning-to-learn model and some useful
and practical tricks. Our optimizer outperforms generic, hand-crafted
optimization algorithms and state-of-the-art learning-to-learn optimizers by
DeepMind in many tasks. We demonstrate the effectiveness of our algorithms on a
number of tasks, including deep MLPs, CNNs, and simple LSTMs.
