Acceleration via Fractal Learning Rate Schedules
  In practical applications of iterative first-order optimization, the learning
rate schedule remains notoriously difficult to understand and expensive to
tune. We demonstrate the presence of these subtleties even in the innocuous
case when the objective is a convex quadratic. We reinterpret an iterative
algorithm from the numerical analysis literature as what we call the Chebyshev
learning rate schedule for accelerating vanilla gradient descent, and show that
the problem of mitigating instability leads to a fractal ordering of step
sizes. We provide some experiments to challenge conventional beliefs about
stable learning rates in deep learning: the fractal schedule enables training
to converge with locally unstable updates which make negative progress on the
objective.
