Tighter Generalization Bounds for Iterative Differentially Private
  Learning Algorithms
  This paper studies the relationship between generalization and privacy
preservation in iterative learning algorithms by two sequential steps. We first
establish an alignment between generalization and privacy preservation for any
learning algorithm. We prove that $(\varepsilon, \delta)$-differential privacy
implies an on-average generalization bound for multi-database learning
algorithms which further leads to a high-probability bound for any learning
algorithm. This high-probability bound also implies a PAC-learnable guarantee
for differentially private learning algorithms. We then investigate how the
iterative nature shared by most learning algorithms influence privacy
preservation and further generalization. Three composition theorems are
proposed to approximate the differential privacy of any iterative algorithm
through the differential privacy of its every iteration. By integrating the
above two steps, we eventually deliver generalization bounds for iterative
learning algorithms, which suggest one can simultaneously enhance privacy
preservation and generalization. Our results are strictly tighter than the
existing works. Particularly, our generalization bounds do not rely on the
model size which is prohibitively large in deep learning. This sheds light to
understanding the generalizability of deep learning. These results apply to a
wide spectrum of learning algorithms. In this paper, we apply them to
stochastic gradient Langevin dynamics and agnostic federated learning as
examples.
