PowerNorm: Rethinking Batch Normalization in Transformers
  The standard normalization method for neural network (NN) models used in
Natural Language Processing (NLP) is layer normalization (LN). This is
different than batch normalization (BN), which is widely-adopted in Computer
Vision. The preferred use of LN in NLP is principally due to the empirical
observation that a (naive/vanilla) use of BN leads to significant performance
degradation for NLP tasks; however, a thorough understanding of the underlying
reasons for this is not always evident. In this paper, we perform a systematic
study of NLP transformer models to understand why BN has a poor performance, as
compared to LN. We find that the statistics of NLP data across the batch
dimension exhibit large fluctuations throughout training. This results in
instability, if BN is naively implemented. To address this, we propose Power
Normalization (PN), a novel normalization scheme that resolves this issue by
(i) relaxing zero-mean normalization in BN, (ii) incorporating a running
quadratic mean instead of per batch statistics to stabilize fluctuations, and
(iii) using an approximate backpropagation for incorporating the running
statistics in the forward pass. We show theoretically, under mild assumptions,
that PN leads to a smaller Lipschitz constant for the loss, compared with BN.
Furthermore, we prove that the approximate backpropagation scheme leads to
bounded gradients. We extensively test PN for transformers on a range of NLP
tasks, and we show that it significantly outperforms both LN and BN. In
particular, PN outperforms LN by 0.4/0.6 BLEU on IWSLT14/WMT14 and 5.6/3.0 PPL
on PTB/WikiText-103. We make our code publicly available at
\url{https://github.com/sIncerass/powernorm}.
