Contrasting random and learned features in deep Bayesian linear
  regression
  Understanding how feature learning affects generalization is among the
foremost goals of modern deep learning theory. Here, we study how the ability
to learn representations affects the generalization performance of a simple
class of models: deep Bayesian linear neural networks trained on unstructured
Gaussian data. By comparing deep random feature models to deep networks in
which all layers are trained, we provide a detailed characterization of the
interplay between width, depth, data density, and prior mismatch. We show that
both models display sample-wise double-descent behavior in the presence of
label noise. Random feature models can also display model-wise double-descent
if there are narrow bottleneck layers, while deep networks do not show these
divergences. Random feature models can have particular widths that are optimal
for generalization at a given data density, while making neural networks as
wide or as narrow as possible is always optimal. Moreover, we show that the
leading-order correction to the kernel-limit learning curve cannot distinguish
between random feature models and deep networks in which all layers are
trained. Taken together, our findings begin to elucidate how architectural
details affect generalization performance in this simple class of deep
regression models.
