Generalization Bounds with Minimal Dependency on Hypothesis Class via
  Distributionally Robust Optimization
  Established approaches to obtain generalization bounds in data-driven
optimization and machine learning mostly build on solutions from empirical risk
minimization (ERM), which depend crucially on the functional complexity of the
hypothesis class. In this paper, we present an alternate route to obtain these
bounds on the solution from distributionally robust optimization (DRO), a
recent data-driven optimization framework based on worst-case analysis and the
notion of ambiguity set to capture statistical uncertainty. In contrast to the
hypothesis class complexity in ERM, our DRO bounds depend on the ambiguity set
geometry and its compatibility with the true loss function. Notably, when using
maximum mean discrepancy as a DRO distance metric, our analysis implies
generalization bounds whose dependence on the hypothesis class appears the
minimal possible: The bound depends solely on the true loss function,
independent of any other candidates in the hypothesis class. To our best
knowledge, it is the first generalization bound of this type in the literature,
and we hope our findings can open the door for a better understanding of DRO,
especially its benefits on loss minimization and other machine learning
applications.
