Characterizing signal propagation to close the performance gap in
  unnormalized ResNets
  Batch Normalization is a key component in almost all state-of-the-art image
classifiers, but it also introduces practical challenges: it breaks the
independence between training examples within a batch, can incur compute and
memory overhead, and often results in unexpected bugs. Building on recent
theoretical analyses of deep ResNets at initialization, we propose a simple set
of analysis tools to characterize signal propagation on the forward pass, and
leverage these tools to design highly performant ResNets without activation
normalization layers. Crucial to our success is an adapted version of the
recently proposed Weight Standardization. Our analysis tools show how this
technique preserves the signal in networks with ReLU or Swish activation
functions by ensuring that the per-channel activation means do not grow with
depth. Across a range of FLOP budgets, our networks attain performance
competitive with the state-of-the-art EfficientNets on ImageNet.
