Memorization-Dilation: Modeling Neural Collapse Under Noise
  The notion of neural collapse refers to several emergent phenomena that have
been empirically observed across various canonical classification problems.
During the terminal phase of training a deep neural network, the feature
embedding of all examples of the same class tend to collapse to a single
representation, and the features of different classes tend to separate as much
as possible. Neural collapse is often studied through a simplified model,
called the unconstrained feature representation, in which the model is assumed
to have "infinite expressivity" and can map each data point to any arbitrary
representation. In this work, we propose a more realistic variant of the
unconstrained feature representation that takes the limited expressivity of the
network into account. Empirical evidence suggests that the memorization of
noisy data points leads to a degradation (dilation) of the neural collapse.
Using a model of the memorization-dilation (M-D) phenomenon, we show one
mechanism by which different losses lead to different performances of the
trained network on noisy data. Our proofs reveal why label smoothing, a
modification of cross-entropy empirically observed to produce a regularization
effect, leads to improved generalization in classification tasks.
