Spectral Analysis of Latent Representations
  We propose a metric, Layer Saturation, defined as the proportion of the
number of eigenvalues needed to explain 99% of the variance of the latent
representations, for analyzing the learned representations of neural network
layers. Saturation is based on spectral analysis and can be computed
efficiently, making live analysis of the representations practical during
training. We provide an outlook for future applications of this metric by
outlining the behaviour of layer saturation in different neural architectures
and problems. We further show that saturation is related to the generalization
and predictive performance of neural networks.
