Reparametrizing gradient descent
  In this work, we propose an optimization algorithm which we call norm-adapted
gradient descent. This algorithm is similar to other gradient-based
optimization algorithms like Adam or Adagrad in that it adapts the learning
rate of stochastic gradient descent at each iteration. However, rather than
using statistical properties of observed gradients, norm-adapted gradient
descent relies on a first-order estimate of the effect of a standard gradient
descent update step, much like the Newton-Raphson method in many dimensions.
Our algorithm can also be compared to quasi-Newton methods, but we seek roots
rather than stationary points. Seeking roots can be justified by the fact that
for models with sufficient capacity measured by nonnegative loss functions,
roots coincide with global optima. This work presents several experiments where
we have used our algorithm; in these results, it appears norm-adapted descent
is particularly strong in regression settings but is also capable of training
classifiers.
