Diagonal Rescaling For Neural Networks
  We define a second-order neural network stochastic gradient training
algorithm whose block-diagonal structure effectively amounts to normalizing the
unit activations. Investigating why this algorithm lacks in robustness then
reveals two interesting insights. The first insight suggests a new way to scale
the stepsizes, clarifying popular algorithms such as RMSProp as well as old
neural network tricks such as fanin stepsize scaling. The second insight
stresses the practical importance of dealing with fast changes of the curvature
of the cost.
