The Spectrum of Fisher Information of Deep Networks Achieving Dynamical
  Isometry
  The Fisher information matrix (FIM) is fundamental to understanding the
trainability of deep neural nets (DNN), since it describes the parameter
space's local metric. We investigate the spectral distribution of the
conditional FIM, which is the FIM given a single sample, by focusing on
fully-connected networks achieving dynamical isometry. Then, while dynamical
isometry is known to keep specific backpropagated signals independent of the
depth, we find that the parameter space's local metric linearly depends on the
depth even under the dynamical isometry. More precisely, we reveal that the
conditional FIM's spectrum concentrates around the maximum and the value grows
linearly as the depth increases. To examine the spectrum, considering random
initialization and the wide limit, we construct an algebraic methodology based
on the free probability theory. As a byproduct, we provide an analysis of the
solvable spectral distribution in two-hidden-layer cases. Lastly, experimental
results verify that the appropriate learning rate for the online training of
DNNs is in inverse proportional to depth, which is determined by the
conditional FIM's spectrum.
