Generalization Through The Lens Of Leave-One-Out Error
  Despite the tremendous empirical success of deep learning models to solve
various learning tasks, our theoretical understanding of their generalization
ability is very limited. Classical generalization bounds based on tools such as
the VC dimension or Rademacher complexity, are so far unsuitable for deep
models and it is doubtful that these techniques can yield tight bounds even in
the most idealistic settings (Nagarajan & Kolter, 2019). In this work, we
instead revisit the concept of leave-one-out (LOO) error to measure the
generalization ability of deep models in the so-called kernel regime. While
popular in statistics, the LOO error has been largely overlooked in the context
of deep learning. By building upon the recently established connection between
neural networks and kernel learning, we leverage the closed-form expression for
the leave-one-out error, giving us access to an efficient proxy for the test
error. We show both theoretically and empirically that the leave-one-out error
is capable of capturing various phenomena in generalization theory, such as
double descent, random labels or transfer learning. Our work therefore
demonstrates that the leave-one-out error provides a tractable way to estimate
the generalization ability of deep neural networks in the kernel regime,
opening the door to potential, new research directions in the field of
generalization.
