Train Feedfoward Neural Network with Layer-wise Adaptive Rate via
  Approximating Back-matching Propagation
  Stochastic gradient descent (SGD) has achieved great success in training deep
neural network, where the gradient is computed through back-propagation.
However, the back-propagated values of different layers vary dramatically. This
inconsistence of gradient magnitude across different layers renders
optimization of deep neural network with a single learning rate problematic. We
introduce the back-matching propagation which computes the backward values on
the layer's parameter and the input by matching backward values on the layer's
output. This leads to solving a bunch of least-squares problems, which requires
high computational cost. We then reduce the back-matching propagation with
approximations and propose an algorithm that turns to be the regular SGD with a
layer-wise adaptive learning rate strategy. This allows an easy implementation
of our algorithm in current machine learning frameworks equipped with
auto-differentiation. We apply our algorithm in training modern deep neural
networks and achieve favorable results over SGD.
