A Theoretical Perspective on Differentially Private Federated Multi-task
  Learning
  In the era of big data, the need to expand the amount of data through data
sharing to improve model performance has become increasingly compelling. As a
result, effective collaborative learning models need to be developed with
respect to both privacy and utility concerns. In this work, we propose a new
federated multi-task learning method for effective parameter transfer with
differential privacy to protect gradients at the client level. Specifically,
the lower layers of the networks are shared across all clients to capture
transferable feature representation, while top layers of the network are
task-specific for on-client personalization. Our proposed algorithm naturally
resolves the statistical heterogeneity problem in federated networks. We are,
to the best of knowledge, the first to provide both privacy and utility
guarantees for such a proposed federated algorithm. The convergences are proved
for the cases with Lipschitz smooth objective functions under the non-convex,
convex, and strongly convex settings. Empirical experiment results on different
datasets have been conducted to demonstrate the effectiveness of the proposed
algorithm and verify the implications of the theoretical findings.
