Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural
  Networks: Orthogonal Case
  The paper deals with the distribution of singular values of the input-output
Jacobian of deep untrained neural networks in the limit of their infinite
width. The Jacobian is the product of random matrices where the independent
rectangular weight matrices alternate with diagonal matrices whose entries
depend on the corresponding column of the nearest neighbor weight matrix. The
problem was considered in \cite{Pe-Co:18} for the Gaussian weights and biases
and also for the weights that are Haar distributed orthogonal matrices and
Gaussian biases. Basing on a free probability argument, it was claimed that in
these cases the singular value distribution of the Jacobian in the limit of
infinite width (matrix size) coincides with that of the analog of the Jacobian
with special random but weight independent diagonal matrices, the case well
known in random matrix theory. The claim was rigorously proved in
\cite{Pa-Sl:21} for a quite general class of weights and biases with i.i.d.
(including Gaussian) entries by using a version of the techniques of random
matrix theory. In this paper we use another version of the techniques to
justify the claim for random Haar distributed weight matrices and Gaussian
biases.
