Exploring the Properties and Evolution of Neural Network Eigenspaces
  during Training
  In this work we explore the information processing inside neural networks
using logistic regression probes \cite{probes} and the saturation metric
\cite{featurespace_saturation}. We show that problem difficulty and neural
network capacity affect the predictive performance in an antagonistic manner,
opening the possibility of detecting over- and under-parameterization of neural
networks for a given task. We further show that the observed effects are
independent from previously reported pathological patterns like the ``tail
pattern'' described in \cite{featurespace_saturation}. Finally we are able to
show that saturation patterns converge early during training, allowing for a
quicker cycle time during analysis
