AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio
  It is well known that we need to choose the hyper-parameters in Momentum,
AdaGrad, AdaDelta, and other alternative stochastic optimizers. While in many
cases, the hyper-parameters are tuned tediously based on experience becoming
more of an art than science. We present a novel per-dimension learning rate
method for gradient descent called AdaSmooth. The method is insensitive to
hyper-parameters thus it requires no manual tuning of the hyper-parameters like
Momentum, AdaGrad, and AdaDelta methods. We show promising results compared to
other methods on different convolutional neural networks, multi-layer
perceptron, and alternative machine learning tasks. Empirical results
demonstrate that AdaSmooth works well in practice and compares favorably to
other stochastic optimization methods in neural networks.
