Altering backward pass gradients improves convergence
  In typical neural network training, the gradients in the backward pass is
determined by the forward pass. As a result, the two stages are coupled.
However, it is often seen that neural networks perform worse when gradients
explode or decline. To address this, numerous approaches like Gradient Clipping
(GC) and Adaptive Gradient Clipping (AGC) have been developed to enhance the
gradient behaviour of networks without normalization layers during backward
passes. These techniques decouple the backward and forward passes and modify
the gradients adaptively. A possible drawback of clipping approaches is that
they must be calculated for each weight tensor in each layer. We offer the
PowerGrad Transform (PGT), a comparable approach that alters and enhances the
gradient flow behaviour in the backward pass but is calculated only in the
final softmax layer. It is very computationally efficient and outperforms both
GC and AGC, resulting in improved performance in networks without batch
normalization. PGT is easy to integrate into existing networks, requiring just
a few lines of code, and significantly increases performance in non-BN ResNets.
The impact is more pronounced on big datasets like as ImageNet, when networks
do not fit all of the training data and there is some training headroom. PGT
makes it possible for the network to better fit the training data while
simultaneously improving its performance on the test set.
