A Sober Look at Neural Network Initializations
  Initializing the weights and the biases is a key part of the training process
of a neural network. Unlike the subsequent optimization phase, however, the
initialization phase has gained only limited attention in the literature. In
this paper we discuss some consequences of commonly used initialization
strategies for vanilla DNNs with ReLU activations. Based on these insights we
then develop an alternative initialization strategy. Finally, we present some
large scale experiments assessing the quality of the new initialization
strategy.
