AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive
  Pruning
  In multi-party collaborative learning, the parameter server sends a global
model to each data holder for local training and then aggregates committed
models globally to achieve privacy protection. However, both the dragger issue
of synchronous collaborative learning and the staleness issue of asynchronous
collaborative learning make collaborative learning inefficient in real-world
heterogeneous environments. We propose a novel and efficient collaborative
learning framework named AdaptCL, which generates an adaptive sub-model
dynamically from the global base model for each data holder, without any prior
information about worker capability. All workers (data holders) achieve
approximately identical update time as the fastest worker by equipping them
with capability-adapted pruned models. Thus the training process can be
dramatically accelerated. Besides, we tailor the efficient pruned rate learning
algorithm and pruning approach for AdaptCL. Meanwhile, AdaptCL provides a
mechanism for handling the trade-off between accuracy and time overhead and can
be combined with other techniques to accelerate training further. Empirical
results show that AdaptCL introduces little computing and communication
overhead. AdaptCL achieves time savings of more than 41\% on average and
improves accuracy in a low heterogeneous environment. In a highly heterogeneous
environment, AdaptCL achieves a training speedup of 6.2x with a slight loss of
accuracy.
