Feature Space Saturation during Training
  We propose layer saturation - a simple, online-computable method for
analyzing the information processing in neural networks. First, we show that a
layer's output can be restricted to the eigenspace of its variance matrix
without performance loss. We propose a computationally lightweight method for
approximating the variance matrix during training. From the dimension of its
lossless eigenspace we derive layer saturation - the ratio between the
eigenspace dimension and layer width. We show that saturation seems to indicate
which layers contribute to network performance. We demonstrate how to alter
layer saturation in a neural network by changing network depth, filter sizes
and input resolution. Furthermore, we show that well-chosen input resolution
increases network performance by distributing the inference process more evenly
across the network.
