Revisiting Initialization of Neural Networks
  The proper initialization of weights is crucial for the effective training
and fast convergence of deep neural networks (DNNs). Prior work in this area
has mostly focused on balancing the variance among weights per layer to
maintain stability of (i) the input data propagated forwards through the
network and (ii) the loss gradients propagated backwards, respectively. This
prevalent heuristic is however agnostic of dependencies among gradients across
the various layers and captures only firstorder effects. In this paper, we
propose and discuss an initialization principle that is based on a rigorous
estimation of the global curvature of weights across layers by approximating
and controlling the norm of their Hessian matrix. The proposed approach is more
systematic and recovers previous results for DNN activations such as smooth
functions, dropouts, and ReLU. Our experiments on Word2Vec and the MNIST/CIFAR
image classification tasks confirm that tracking the Hessian norm is a useful
diagnostic tool which helps to more rigorously initialize weights
