Almost Sure Asymptotic Freeness of Neural Network Jacobian with
  Orthogonal Weights
  A well-conditioned Jacobian spectrum has a vital role in preventing exploding
or vanishing gradients and speeding up learning of deep neural networks. Free
probability theory helps us to understand and handle the Jacobian spectrum. We
rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep
neural networks as the wide limit. In particular, we treat the case that
weights are initialized as Haar distributed orthogonal matrices.
