Resurrecting the sigmoid in deep learning through dynamical isometry:
  theory and practice
  It is well known that the initialization of weights in deep neural networks
can have a dramatic impact on learning speed. For example, ensuring the mean
squared singular value of a network's input-output Jacobian is $O(1)$ is
essential for avoiding the exponential vanishing or explosion of gradients. The
stronger condition that all singular values of the Jacobian concentrate near
$1$ is a property known as dynamical isometry. For deep linear networks,
dynamical isometry can be achieved through orthogonal weight initialization and
has been shown to dramatically speed up learning; however, it has remained
unclear how to extend these results to the nonlinear setting. We address this
question by employing powerful tools from free probability theory to compute
analytically the entire singular value distribution of a deep network's
input-output Jacobian. We explore the dependence of the singular value
distribution on the depth of the network, the weight initialization, and the
choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable
of dynamical isometry. On the other hand, sigmoidal networks can achieve
isometry, but only with orthogonal weight initialization. Moreover, we
demonstrate empirically that deep nonlinear networks achieving dynamical
isometry learn orders of magnitude faster than networks that do not. Indeed, we
show that properly-initialized deep sigmoidal networks consistently outperform
deep ReLU networks. Overall, our analysis reveals that controlling the entire
distribution of Jacobian singular values is an important design consideration
in deep learning.
