Neural Tangent Kernel Empowered Federated Learning
  Federated learning (FL) is a privacy-preserving paradigm where multiple
participants jointly solve a machine learning problem without sharing raw data.
Unlike traditional distributed learning, a unique characteristic of FL is
statistical heterogeneity, namely, data distributions across participants are
different from each other. Meanwhile, recent advances in the interpretation of
neural networks have seen a wide use of neural tangent kernels (NTKs) for
convergence analyses. In this paper, we propose a novel FL paradigm empowered
by the NTK framework. The paradigm addresses the challenge of statistical
heterogeneity by transmitting update data that are more expressive than those
of the conventional FL paradigms. Specifically, sample-wise Jacobian matrices,
rather than model weights/gradients, are uploaded by participants. The server
then constructs an empirical kernel matrix to update a global model without
explicitly performing gradient descent. We further develop a variant with
improved communication efficiency and enhanced privacy. Numerical results show
that the proposed paradigm can achieve the same accuracy while reducing the
number of communication rounds by an order of magnitude compared to federated
averaging.
