Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft
  Committee Machines
  Over-parametrized deep neural networks trained by stochastic gradient descent
are successful in performing many tasks of practical relevance. One aspect of
over-parametrization is the possibility that the student network has a larger
expressivity than the data generating process. In the context of a
student-teacher scenario, this corresponds to the so-called over-realizable
case, where the student network has a larger number of hidden units than the
teacher. For on-line learning of a two-layer soft committee machine in the
over-realizable case, we find that the approach to perfect learning occurs in a
power-law fashion rather than exponentially as in the realizable case. All
student nodes learn and replicate one of the teacher nodes if teacher and
student outputs are suitably rescaled.
