More Than a Toy: Random Matrix Models Predict How Real-World Neural
  Representations Generalize
  Of theories for why large-scale machine learning models generalize despite
being vastly overparameterized, which of their assumptions are needed to
capture the qualitative phenomena of generalization in the real world? On one
hand, we find that most theoretical analyses fall short of capturing these
qualitative phenomena even for kernel regression, when applied to kernels
derived from large-scale neural networks (e.g., ResNet-50) and real data (e.g.,
CIFAR-100). On the other hand, we find that the classical GCV estimator (Craven
and Wahba, 1978) accurately predicts generalization risk even in such
overparameterized settings. To bolster this empirical finding, we prove that
the GCV estimator converges to the generalization risk whenever a local random
matrix law holds. Finally, we apply this random matrix theory lens to explain
why pretrained representations generalize better as well as what factors govern
scaling laws for kernel regression. Our findings suggest that random matrix
theory, rather than just being a toy model, may be central to understanding the
properties of neural representations in practice.
