Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of
  Multilayer Perceptron: The Haar Orthogonal Case
  Free Probability Theory (FPT) provides rich knowledge for handling
mathematical difficulties caused by random matrices that appear in research
related to deep neural networks (DNNs), such as the dynamical isometry, Fisher
information matrix, and training dynamics. FPT suits these researches because
the DNN's parameter-Jacobian and input-Jacobian are polynomials of layerwise
Jacobians. However, the critical assumption of asymptotic freenss of the
layerwise Jacobian has not been proven completely so far. The asymptotic
freeness assumption plays a fundamental role when propagating spectral
distributions through the layers. Haar distributed orthogonal matrices are
essential for achieving dynamical isometry. In this work, we prove asymptotic
freeness of layerwise Jacobians of multilayer perceptron (MLP) in this case. A
key of the proof is an invariance of the MLP. Considering the orthogonal
matrices that fix the hidden units in each layer, we replace each layer's
parameter matrix with itself multiplied by the orthogonal matrix, and then the
MLP does not change. Furthermore, if the original weights are Haar orthogonal,
the Jacobian is also unchanged by this replacement. Lastly, we can replace each
weight with a Haar orthogonal random matrix independent of the Jacobian of the
activation function using this key fact.
