Rethinking Parameter Counting in Deep Models: Effective Dimensionality
  Revisited
  Neural networks appear to have mysterious generalization properties when
using parameter counting as a proxy for complexity. Indeed, neural networks
often have many more parameters than there are data points, yet still provide
good generalization performance. Moreover, when we measure generalization as a
function of parameters, we see double descent behaviour, where the test error
decreases, increases, and then again decreases. We show that many of these
properties become understandable when viewed through the lens of effective
dimensionality, which measures the dimensionality of the parameter space
determined by the data. We relate effective dimensionality to posterior
contraction in Bayesian deep learning, model selection, width-depth tradeoffs,
double descent, and functional diversity in loss surfaces, leading to a richer
understanding of the interplay between parameters and functions in deep models.
We also show that effective dimensionality compares favourably to alternative
norm- and flatness- based generalization measures.
