Adaptive Learning Rate Clipping Stabilizes Learning
  Artificial neural network training with stochastic gradient descent can be
destabilized by "bad batches" with high losses. This is often problematic for
training with small batch sizes, high order loss functions or unstably high
learning rates. To stabilize learning, we have developed adaptive learning rate
clipping (ALRC) to limit backpropagated losses to a number of standard
deviations above their running means. ALRC is designed to complement existing
learning algorithms: Our algorithm is computationally inexpensive, can be
applied to any loss function or batch size, is robust to hyperparameter choices
and does not affect backpropagated gradient distributions. Experiments with
CIFAR-10 supersampling show that ALCR decreases errors for unstable mean
quartic error training while stable mean squared error training is unaffected.
We also show that ALRC decreases unstable mean squared errors for partial
scanning transmission electron micrograph completion. Our source code is
publicly available at https://github.com/Jeffrey-Ede/ALRC
