Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style
  Adaptive Momentum
  Momentum plays a crucial role in stochastic gradient-based optimization
algorithms for accelerating or improving training deep neural networks (DNNs).
In deep learning practice, the momentum is usually weighted by a
well-calibrated constant. However, tuning hyperparameters for momentum can be a
significant computational burden. In this paper, we propose a novel
\emph{adaptive momentum} for improving DNNs training; this adaptive momentum,
with no momentum related hyperparameter required, is motivated by the nonlinear
conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this
new adaptive momentum eliminates the need for the momentum hyperparameter
calibration, allows a significantly larger learning rate, accelerates DNN
training, and improves final accuracy and robustness of the trained DNNs. For
instance, SGD with this adaptive momentum reduces classification errors for
training ResNet110 for CIFAR10 and CIFAR100 from $5.25\%$ to $4.64\%$ and
$23.75\%$ to $20.03\%$, respectively. Furthermore, SGD with the new adaptive
momentum also benefits adversarial training and improves adversarial robustness
of the trained DNNs.
