Disentangling Adaptive Gradient Methods from Learning Rates
  We investigate several confounding factors in the evaluation of optimization
algorithms for deep learning. Primarily, we take a deeper look at how adaptive
gradient methods interact with the learning rate schedule, a notoriously
difficult-to-tune hyperparameter which has dramatic effects on the convergence
and generalization of neural network training. We introduce a "grafting"
experiment which decouples an update's magnitude from its direction, finding
that many existing beliefs in the literature may have arisen from insufficient
isolation of the implicit schedule of step sizes. Alongside this contribution,
we present some empirical and theoretical retrospectives on the generalization
of adaptive gradient methods, aimed at bringing more clarity to this space.
