A Geometric Analysis of Neural Collapse with Unconstrained Features
  We provide the first global optimization landscape analysis of
$Neural\;Collapse$ -- an intriguing empirical phenomenon that arises in the
last-layer classifiers and features of neural networks during the terminal
phase of training. As recently reported by Papyan et al., this phenomenon
implies that ($i$) the class means and the last-layer classifiers all collapse
to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and
($ii$) cross-example within-class variability of last-layer activations
collapses to zero. We study the problem based on a simplified
$unconstrained\;feature\;model$, which isolates the topmost layers from the
classifier of the neural network. In this context, we show that the classical
cross-entropy loss with weight decay has a benign global landscape, in the
sense that the only global minimizers are the Simplex ETFs while all other
critical points are strict saddles whose Hessian exhibit negative curvature
directions. In contrast to existing landscape analysis for deep neural networks
which is often disconnected from practice, our analysis of the simplified model
not only does it explain what kind of features are learned in the last layer,
but it also shows why they can be efficiently optimized in the simplified
settings, matching the empirical observations in practical deep network
architectures. These findings could have profound implications for
optimization, generalization, and robustness of broad interests. For example,
our experiments demonstrate that one may set the feature dimension equal to the
number of classes and fix the last-layer classifier to be a Simplex ETF for
network training, which reduces memory cost by over $20\%$ on ResNet18 without
sacrificing the generalization performance.
