Redundant representations help generalization in wide neural networks
  Deep neural networks (DNNs) defy the classical bias-variance trade-off:
adding parameters to a DNN that interpolates its training data will typically
improve its generalization performance. Explaining the mechanism behind this
``benign overfitting'' in deep networks remains an outstanding challenge. Here,
we study the last hidden layer representations of various state-of-the-art
convolutional neural networks and find that if the last hidden representation
is wide enough, its neurons tend to split into groups that carry identical
information, and differ from each other only by statistically independent
noise. The number of such groups increases linearly with the width of the
layer, but only if the width is above a critical value. We show that redundant
neurons appear only when the training process reaches interpolation and the
training error is zero.
