Stability Based Generalization Bounds for Exponential Family Langevin
  Dynamics
  We study generalization bounds for noisy stochastic mini-batch iterative
algorithms based on the notion of stability. Recent years have seen key
advances in data-dependent generalization bounds for noisy iterative learning
algorithms such as stochastic gradient Langevin dynamics (SGLD) based on
stability (Mou et al., 2018; Li et al., 2020) and information theoretic
approaches (Xu and Raginsky, 2017; Negrea et al., 2019; Steinke and
Zakynthinou, 2020; Haghifam et al., 2020). In this paper, we unify and
substantially generalize stability based generalization bounds and make three
technical advances. First, we bound the generalization error of general noisy
stochastic iterative algorithms (not necessarily gradient descent) in terms of
expected (not uniform) stability. The expected stability can in turn be bounded
by a Le Cam Style Divergence. Such bounds have a O(1/n) sample dependence
unlike many existing bounds with O(1/\sqrt{n}) dependence. Second, we introduce
Exponential Family Langevin Dynamics(EFLD) which is a substantial
generalization of SGLD and which allows exponential family noise to be used
with stochastic gradient descent (SGD). We establish data-dependent expected
stability based generalization bounds for general EFLD algorithms. Third, we
consider an important special case of EFLD: noisy sign-SGD, which extends
sign-SGD using Bernoulli noise over {-1,+1}. Generalization bounds for noisy
sign-SGD are implied by that of EFLD and we also establish optimization
guarantees for the algorithm. Further, we present empirical results on
benchmark datasets to illustrate that our bounds are non-vacuous and
quantitatively much sharper than existing bounds.
