Towards Optimal Problem Dependent Generalization Error Bounds in
  Statistical Learning Theory
  We study problem-dependent rates, i.e., generalization errors that scale
near-optimally with the variance, the effective loss, or the gradient norms
evaluated at the "best hypothesis." We introduce a principled framework dubbed
"uniform localized convergence," and characterize sharp problem-dependent rates
for central statistical learning problems. From a methodological viewpoint, our
framework resolves several fundamental limitations of existing uniform
convergence and localization analysis approaches. It also provides improvements
and some level of unification in the study of localized complexities, one-sided
uniform inequalities, and sample-based iterative algorithms. In the so-called
"slow rate" regime, we provides the first (moment-penalized) estimator that
achieves the optimal variance-dependent rate for general "rich" classes; we
also establish improved loss-dependent rate for standard empirical risk
minimization. In the "fast rate" regime, we establish finite-sample
problem-dependent bounds that are comparable to precise asymptotics. In
addition, we show that iterative algorithms like gradient descent and
first-order Expectation-Maximization can achieve optimal generalization error
in several representative problems across the areas of non-convex learning,
stochastic optimization, and learning with missing data.
