Uniform Generalization, Concentration, and Adaptive Learning
  One fundamental goal in any learning algorithm is to mitigate its risk for
overfitting. Mathematically, this requires that the learning algorithm enjoys a
small generalization risk, which is defined either in expectation or in
probability. Both types of generalization are commonly used in the literature.
For instance, generalization in expectation has been used to analyze
algorithms, such as ridge regression and SGD, whereas generalization in
probability is used in the VC theory, among others. Recently, a third notion of
generalization has been studied, called uniform generalization, which requires
that the generalization risk vanishes uniformly in expectation across all
bounded parametric losses. It has been shown that uniform generalization is, in
fact, equivalent to an information-theoretic stability constraint, and that it
recovers classical results in learning theory. It is achievable under various
settings, such as sample compression schemes, finite hypothesis spaces, finite
domains, and differential privacy. However, the relationship between uniform
generalization and concentration remained unknown. In this paper, we answer
this question by proving that, while a generalization in expectation does not
imply a generalization in probability, a uniform generalization in expectation
does imply concentration. We establish a chain rule for the uniform
generalization risk of the composition of hypotheses and use it to derive a
large deviation bound. Finally, we prove that the bound is tight.
