Improved Information Theoretic Generalization Bounds for Distributed and
  Federated Learning
  We consider information-theoretic bounds on expected generalization error for
statistical learning problems in a networked setting. In this setting, there
are $K$ nodes, each with its own independent dataset, and the models from each
node have to be aggregated into a final centralized model. We consider both
simple averaging of the models as well as more complicated multi-round
algorithms. We give upper bounds on the expected generalization error for a
variety of problems, such as those with Bregman divergence or Lipschitz
continuous losses, that demonstrate an improved dependence of $1/K$ on the
number of nodes. These "per node" bounds are in terms of the mutual information
between the training dataset and the trained weights at each node, and are
therefore useful in describing the generalization properties inherent to having
communication or privacy constraints at each node.
