The learning phases in NN: From Fitting the Majority to Fitting a Few
  The learning dynamics of deep neural networks are subject to controversy.
Using the information bottleneck (IB) theory separate fitting and compression
phases have been put forward but have since been heavily debated. We approach
learning dynamics by analyzing a layer's reconstruction ability of the input
and prediction performance based on the evolution of parameters during
training. We show that a prototyping phase decreasing reconstruction loss
initially, followed by reducing classification loss of a few samples, which
increases reconstruction loss, exists under mild assumptions on the data. Aside
from providing a mathematical analysis of single layer classification networks,
we also assess the behavior using common datasets and architectures from
computer vision such as ResNet and VGG.
