Explaining Neural Scaling Laws
  The test loss of well-trained neural networks often follows precise power-law
scaling relations with either the size of the training dataset or the number of
parameters in the network. We propose a theory that explains and connects these
scaling laws. We identify variance-limited and resolution-limited scaling
behavior for both dataset and model size, for a total of four scaling regimes.
The variance-limited scaling follows simply from the existence of a
well-behaved infinite data or infinite width limit, while the
resolution-limited regime can be explained by positing that models are
effectively resolving a smooth data manifold. In the large width limit, this
can be equivalently obtained from the spectrum of certain kernels, and we
present evidence that large width and large dataset resolution-limited scaling
exponents are related by a duality. We exhibit all four scaling regimes in the
controlled setting of large random feature and pretrained models and test the
predictions empirically on a range of standard architectures and datasets. We
also observe several empirical relationships between datasets and scaling
exponents: super-classing image tasks does not change exponents, while changing
input distribution (via changing datasets or adding noise) has a strong effect.
We further explore the effect of architecture aspect ratio on scaling
exponents.
