Random matrix analysis of deep neural network weight matrices
  Neural networks have been used successfully in a variety of fields, which has
led to a great deal of interest in developing a theoretical understanding of
how they store the information needed to perform a particular task. We study
the weight matrices of trained deep neural networks using methods from random
matrix theory (RMT) and show that the statistics of most of the singular values
follow universal RMT predictions. This suggests that they are random and do not
contain system specific information, which we investigate further by comparing
the statistics of eigenvector entries to the universal Porter-Thomas
distribution. We find that for most eigenvectors the hypothesis of randomness
cannot be rejected, and that only eigenvectors belonging to the largest
singular values deviate from the RMT prediction, indicating that they may
encode learned information. We analyze the spectral distribution of such large
singular values using the Hill estimator and find that the distribution cannot
be characterized by a tail index, i.e. is not of power law type.
