Pathological spectra of the Fisher information metric and its variants
  in deep neural networks
  The Fisher information matrix (FIM) plays an essential role in statistics and
machine learning as a Riemannian metric tensor or a component of the Hessian
matrix of loss functions. Focusing on the FIM and its variants in deep neural
networks (DNNs), we reveal their characteristic scale dependence on the network
width, depth and sample size when the network has random weights and is
sufficiently wide. This study covers two widely-used FIMs for regression with
linear output and for classification with softmax output. Both FIMs
asymptotically show pathological eigenvalue spectra in the sense that a small
number of eigenvalues become large outliers depending the width or sample size
while the others are much smaller. It implies that the local shape of the
parameter space or loss landscape is very sharp in a few specific directions
while almost flat in the other directions. In particular, the softmax output
disperses the outliers and makes a tail of the eigenvalue density spread from
the bulk. We also show that pathological spectra appear in other variants of
FIMs: one is the neural tangent kernel; another is a metric for the input
signal and feature space that arises from feedforward signal propagation. Thus,
we provide a unified perspective on the FIM and its variants that will lead to
more quantitative understanding of learning in large-scale DNNs.
