Rethinking Skip Connection with Layer Normalization in Transformers and
  ResNets
  Skip connection, is a widely-used technique to improve the performance and
the convergence of deep neural networks, which is believed to relieve the
difficulty in optimization due to non-linearity by propagating a linear
component through the neural network layers. However, from another point of
view, it can also be seen as a modulating mechanism between the input and the
output, with the input scaled by a pre-defined value one. In this work, we
investigate how the scale factors in the effectiveness of the skip connection
and reveal that a trivial adjustment of the scale will lead to spurious
gradient exploding or vanishing in line with the deepness of the models, which
could be addressed by normalization, in particular, layer normalization, which
induces consistent improvements over the plain skip connection. Inspired by the
findings, we further propose to adaptively adjust the scale of the input by
recursively applying skip connection with layer normalization, which promotes
the performance substantially and generalizes well across diverse tasks
including both machine translation and image classification datasets.
