Spectrum concentration in deep residual learning: a free probability
  approach
  We revisit the initialization of deep residual networks (ResNets) by
introducing a novel analytical tool in free probability to the community of
deep learning. This tool deals with non-Hermitian random matrices, rather than
their conventional Hermitian counterparts in the literature. As a consequence,
this new tool enables us to evaluate the singular value spectrum of the
input-output Jacobian of a fully-connected deep ResNet for both linear and
nonlinear cases. With the powerful tool of free probability, we conduct an
asymptotic analysis of the spectrum on the single-layer case, and then extend
this analysis to the multi-layer case of an arbitrary number of layers. In
particular, we propose to rescale the classical random initialization by the
number of residual units, so that the spectrum has the order of $O(1)$, when
compared with the large width and depth of the network. We empirically
demonstrate that the proposed initialization scheme learns at a speed of orders
of magnitudes faster than the classical ones, and thus attests a strong
practical relevance of this investigation.
