Dyna: A Method of Momentum for Stochastic Optimization
  An algorithm is presented for momentum gradient descent optimization based on
the first-order differential equation of the Newtonian dynamics. The fictitious
mass is introduced to the dynamics of momentum for regularizing the adaptive
stepsize of each individual parameter. The dynamic relaxation is adapted for
stochastic optimization of nonlinear objective functions through an explicit
time integration with varying damping ratio. The adaptive stepsize is optimized
for each individual neural network layer based on the number of inputs. The
adaptive stepsize for every parameter over the entire neural network is
uniformly optimized with one upper bound, independent of sparsity, for better
overall convergence rate. The numerical implementation of the algorithm is
similar to the Adam Optimizer, possessing computational efficiency, similar
memory requirements, etc. There are three hyper-parameters in the algorithm
with clear physical interpretation. Preliminary trials show promise in
performance and convergence.
