Server-Side Local Gradient Averaging and Learning Rate Acceleration for
  Scalable Split Learning
  In recent years, there have been great advances in the field of decentralized
learning with private data. Federated learning (FL) and split learning (SL) are
two spearheads possessing their pros and cons, and are suited for many user
clients and large models, respectively. To enjoy both benefits, hybrid
approaches such as SplitFed have emerged of late, yet their fundamentals have
still been illusive. In this work, we first identify the fundamental
bottlenecks of SL, and thereby propose a scalable SL framework, coined SGLR.
The server under SGLR broadcasts a common gradient averaged at the split-layer,
emulating FL without any additional communication across clients as opposed to
SplitFed. Meanwhile, SGLR splits the learning rate into its server-side and
client-side rates, and separately adjusts them to support many clients in
parallel. Simulation results corroborate that SGLR achieves higher accuracy
than other baseline SL methods including SplitFed, which is even on par with FL
consuming higher energy and communication costs. As a secondary result, we
observe greater reduction in leakage of sensitive information via mutual
information using SLGR over the baselines.
