Approximate Spectral Decomposition of Fisher Information Matrix for
  Simple ReLU Networks
  We argue the Fisher information matrix (FIM) of one hidden layer networks
with the ReLU activation function. Let $W$ denote the $d \times p$ weight
matrix from the $d$-dimensional input to the hidden layer consisting of $p$
neurons, and $v$ the $p$-dimensional weight vector from the hidden layer to the
scalar output. We focus on the FIM of $v$, which we denote as $I$. When $p$ is
large, under certain conditions, the following approximately holds. 1) There
are three major clusters in the eigenvalue distribution. 2) Since $I$ is
non-negative owing to the ReLU, the first eigenvalue is the Perron-Frobenius
eigenvalue. 3) For the cluster of the next maximum values, the eigenspace is
spanned by the row vectors of $W$. 4) The direct sum of the eigenspace of the
first eigenvalue and that of the third cluster is spanned by the set of all the
vectors obtained as the Hadamard product of any pair of the row vectors of $W$.
We confirmed by numerical simulation that the above is approximately correct
when the number of hidden nodes is about 10000.
