Generalization Error for Linear Regression under Distributed Learning
  Distributed learning facilitates the scaling-up of data processing by
distributing the computational burden over several nodes. Despite the vast
interest in distributed learning, generalization performance of such approaches
is not well understood. We address this gap by focusing on a linear regression
setting. We consider the setting where the unknowns are distributed over a
network of nodes. We present an analytical characterization of the dependence
of the generalization error on the partitioning of the unknowns over nodes. In
particular, for the overparameterized case, our results show that while the
error on training data remains in the same range as that of the centralized
solution, the generalization error of the distributed solution increases
dramatically compared to that of the centralized solution when the number of
unknowns estimated at any node is close to the number of observations. We
further provide numerical examples to verify our analytical expressions.
