AsySQN: Faster Vertical Federated Learning Algorithms with Better
  Computation Resource Utilization
  Vertical federated learning (VFL) is an effective paradigm of training the
emerging cross-organizational (e.g., different corporations, companies and
organizations) collaborative learning with privacy preserving. Stochastic
gradient descent (SGD) methods are the popular choices for training VFL models
because of the low per-iteration computation. However, existing SGD-based VFL
algorithms are communication-expensive due to a large number of communication
rounds. Meanwhile, most existing VFL algorithms use synchronous computation
which seriously hamper the computation resource utilization in real-world
applications. To address the challenges of communication and computation
resource utilization, we propose an asynchronous stochastic quasi-Newton
(AsySQN) framework for VFL, under which three algorithms, i.e. AsySQN-SGD,
-SVRG and -SAGA, are proposed. The proposed AsySQN-type algorithms making
descent steps scaled by approximate (without calculating the inverse Hessian
matrix explicitly) Hessian information convergence much faster than SGD-based
methods in practice and thus can dramatically reduce the number of
communication rounds. Moreover, the adopted asynchronous computation can make
better use of the computation resource. We theoretically prove the convergence
rates of our proposed algorithms for strongly convex problems. Extensive
numerical experiments on real-word datasets demonstrate the lower communication
costs and better computation resource utilization of our algorithms compared
with state-of-the-art VFL algorithms.
