Proxy-Normalizing Activations to Match Batch Normalization while
  Removing Batch Dependence
  We investigate the reasons for the performance degradation incurred with
batch-independent normalization. We find that the prototypical techniques of
layer normalization and instance normalization both induce the appearance of
failure modes in the neural network's pre-activations: (i) layer normalization
induces a collapse towards channel-wise constant functions; (ii) instance
normalization induces a lack of variability in instance statistics, symptomatic
of an alteration of the expressivity. To alleviate failure mode (i) without
aggravating failure mode (ii), we introduce the technique "Proxy Normalization"
that normalizes post-activations using a proxy distribution. When combined with
layer normalization or group normalization, this batch-independent
normalization emulates batch normalization's behavior and consistently matches
or exceeds its performance.
