DEAM: Adaptive Momentum with Discriminative Weight for Stochastic
  Optimization
  Optimization algorithms with momentum, e.g., (ADAM), have been widely used
for building deep learning models due to the faster convergence rates compared
with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the
relevant directions in parameter updating, which can minify the oscillations of
parameters update route. However, there exist errors in some update steps in
optimization algorithms with momentum like ADAM. The fixed momentum weight
(e.g., \beta_1 in ADAM) will propagate errors in momentum computing. In this
paper, we introduce a novel optimization algorithm, namely Discriminative
wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term
weight with a fixed hyperparameter, DEAM proposes to compute the momentum
weight automatically based on the discriminative angle. In this way, DEAM
involves fewer hyperparameters. DEAM also contains a novel backtrack term,
which restricts redundant updates when the correction of the last step is
needed. Extensive experiments demonstrate that DEAM can achieve a faster
convergence rate than the existing optimization algorithms in training the deep
learning models of both convex and non-convex situations.
