Proportionate gradient updates with PercentDelta
  Deep Neural Networks are generally trained using iterative gradient updates.
Magnitudes of gradients are affected by many factors, including choice of
activation functions and initialization. More importantly, gradient magnitudes
can greatly differ across layers, with some layers receiving much smaller
gradients than others. causing some layers to train slower than others and
therefore slowing down the overall convergence. We analytically explain this
disproportionality. Then we propose to explicitly train all layers at the same
speed, by scaling the gradient w.r.t. every trainable tensor to be proportional
to its current value. In particular, at every batch, we want to update all
trainable tensors, such that the relative change of the L1-norm of the tensors
is the same, across all layers of the network, throughout training time.
Experiments on MNIST show that our method appropriately scales gradients, such
that the relative change in trainable tensors is approximately equal across
layers. In addition, measuring the test accuracy with training time, shows that
our method trains faster than other methods, giving higher test accuracy given
same budget of training steps.
