How to Initialize your Network? Robust Initialization for WeightNorm &
  ResNets
  Residual networks (ResNet) and weight normalization play an important role in
various deep learning applications. However, parameter initialization
strategies have not been studied previously for weight normalized networks and,
in practice, initialization methods designed for un-normalized networks are
used as a proxy. Similarly, initialization for ResNets have also been studied
for un-normalized networks and often under simplified settings ignoring the
shortcut connection. To address these issues, we propose a novel parameter
initialization strategy that avoids explosion/vanishment of information across
layers for weight normalized networks with and without residual connections.
The proposed strategy is based on a theoretical analysis using mean field
approximation. We run over 2,500 experiments and evaluate our proposal on image
datasets showing that the proposed initialization outperforms existing
initialization methods in terms of generalization performance, robustness to
hyper-parameter values and variance between seeds, especially when networks get
deeper in which case existing methods fail to even start training. Finally, we
show that using our initialization in conjunction with learning rate warmup is
able to reduce the gap between the performance of weight normalized and batch
normalized networks.
