Linear Regression with Distributed Learning: A Generalization Error
  Perspective
  Distributed learning provides an attractive framework for scaling the
learning task by sharing the computational load over multiple nodes in a
network. Here, we investigate the performance of distributed learning for
large-scale linear regression where the model parameters, i.e., the unknowns,
are distributed over the network. We adopt a statistical learning approach. In
contrast to works that focus on the performance on the training data, we focus
on the generalization error, i.e., the performance on unseen data. We provide
high-probability bounds on the generalization error for both isotropic and
correlated Gaussian data as well as sub-gaussian data. These results reveal the
dependence of the generalization performance on the partitioning of the model
over the network. In particular, our results show that the generalization error
of the distributed solution can be substantially higher than that of the
centralized solution even when the error on the training data is at the same
level for both the centralized and distributed approaches. Our numerical
results illustrate the performance with both real-world image data as well as
synthetic data.
