Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
  In several recently proposed stochastic optimization methods (e.g. RMSProp,
Adam, Adadelta), parameter updates are scaled by the inverse square roots of
exponential moving averages of squared past gradients. Maintaining these
per-parameter second-moment estimators requires memory equal to the number of
parameters. For the case of neural network weight matrices, we propose
maintaining only the per-row and per-column sums of these moving averages, and
estimating the per-parameter second moments based on these sums. We demonstrate
empirically that this method produces similar results to the baseline.
Secondly, we show that adaptive methods can produce larger-than-desired updates
when the decay rate of the second moment accumulator is too slow. We propose
update clipping and a gradually increasing decay rate scheme as remedies.
Combining these methods and dropping momentum, we achieve comparable results to
the published Adam regime in training the Transformer model on the WMT 2014
English-German machine translation task, while using very little auxiliary
storage in the optimizer. Finally, we propose scaling the parameter updates
based on the scale of the parameters themselves.
